Optiver logs every agent tool call, but re-running old cases is still imperfect

Noyan Tokgozoglu leads AI engineering at the trading firm Optiver. On the AI Engineer podcast, he called testing AI agents against past runs an open problem. Optiver records every tool call. Even so, re-running an old case with a different model is imperfect, because the world the tools saw has changed.

What was said

Tokgozoglu said Optiver logs every tool call, with its inputs and outputs, in a standard logging format called OTEL. That record feeds the firm's evaluation system. The trouble starts when the team wants to run an agent again with a different model.

His example was a support bot that runs in Slack. Its tools depend on the state of the world, and that state was different when the original run happened. So the re-run is not a like-for-like test.

Here is a simple picture, not his words. Say a tool reads a channel. Today it returns different messages than it did then. The new model would be judged on facts the original run never saw.

Optiver tries to snapshot that state. For a trading backtest database, he said, this is easy, because you can always point to a past moment. Slack is harder. He asked what Slack looked like when the agent first ran. There may be ways to replay Slack from a past point, he said, but other systems, not really.

So the team records a confidence value for each tool call. It shows how well that call can be pinned to its original moment, and it is logged with the call. On a re-run, a proxy around the firm's MCP connections handles the snapshot. MCP is a common way for agents to reach tools. His verdict: "it's imperfect and I don't know if there's a perfect solution other than just record everything."

Why it matters

Our reading: a full log is a strong start, but this talk shows its limit. A log records what happened. It cannot always bring back the facts the agent saw.

That matters for anyone who buys or approves agents. Swapping in a new model means comparing it with old runs. Where the re-run is shaky, the comparison is shaky too.

A fair question for any vendor or internal team is which tools can be pinned to a past moment and which cannot. We would also add a point he did not make. If a failed run cannot be replayed faithfully, finding its cause is harder, and it is less clear who answers for the cost.

The other side

Tokgozoglu's talk was about evaluating agents with a different model. He did not discuss replaying failures or incidents. Applying his point to failures is our own extension.

His case is also not hopeless. He said the trading backtest database is easy to query at a past point in time. Slack and other systems are harder.

He did not claim a fix. Confidence values and snapshots are a partial workaround, and he said he does not know of a perfect solution. He also did not say who pays when an agent fails, so the question of ownership stays open.