Optiver found some fresh AI sessions starting with 200k tokens of context

Noyan Tokgozoglu, head of AI Engineering at the trading firm Optiver, described how fully self-service AI use produced waste at his firm. He spoke on the AI Engineer show. Some users opened a fresh session already carrying about 200k tokens of context. Optiver responded with tool search, cache changes and knowledge bases, which he said helped bring spend under control. Separately, showing teams their spend helped them spend less.

What was said

Optiver wants its traders, whom Tokgozoglu called high-agency people, to serve themselves. He said purely self-service tools can turn chaotic, so the firm also offers good defaults.

An inference gateway lets Optiver measure spend on each model and assign budgets. After the big frontier model releases of the summer, he said, spend shot up. Optimizations then brought it back under control. He did not say how much of that spike came from context bloat.

The bloat was a separate discovery. Optiver has many MCPs, which are connectors that give agents tools, plus many knowledge bases. On a fresh session, he said, people sometimes carried about 200k tokens of context by default, occasionally more. Tokens are the chunks of text a model reads and bills for. He called this practice, "That's ridiculous, right?"

Optiver responded with tool search, cache changes and moving material from oversized system prompts into knowledge bases. He said these helped bring spend under control, and gave no before-and-after figure for the starting context. He added that these savings grow as token prices rise.

The second lever was visibility. Tokgozoglu said that when people are told what they spend, and get alerts and the freedom to self-manage, "they actually do a pretty good job." Teams set their own limits, and he said many set reasonable ones.

Why it matters

Our reading: AI cost risk is not only about models making mistakes. It also comes from sprawl, where each user adds one more tool, document or prompt and nobody sees the total. That is an old weakness. AI makes it costlier because every token is billed.

Managers who sign the budget can ask two questions. What does a brand-new session load before any work starts? And can each team see its own spend, with alerts it controls?

The other side

This is one firm's account. He gave no figure for how far spending fell, so the size of the saving is unknown. He said many teams set reasonable limits, which is not all of them.

Optiver is also unusual. It runs its AI platform on its own servers, has a sizeable AI engineering team, and sees the money result of changes within hours. Many firms have none of that.