- Laurie Voss showed that a newer model passed a consistency check by giving the same answer nearly every time, while costing four times as much.
- Without evals, a team could mistake that result for progress. With them, he could show what changed and what got worse.
A model upgrade can look like progress while making an AI system worse. Laurie Voss of Arize AI made this point on the AI Engineer show. His example was a financial analyst agent built on Claude. The newer model passed a consistency check, but only by giving the same answer nearly every time. It also cost four times as much.
What was said
Voss's agent writes stock reports and makes a buy, sell or hold call. He ran it on Haiku 4.5 and on the newer Haiku 5.5, then compared the scores. Evals are automated tests that score what the agent produces. A full run of three repetitions takes about 13 minutes, so he showed results that were already run.
The results were mixed. Faithfulness, meaning how well reports stick to the facts, got more than three times better. Voss still called the agent terrible, because it kept making things up. Consistency also improved: the agent made the same call across repeated runs.
The reason was the problem. The newer model stopped switching between buy and sell and became more conservative. Voss said: "Haiku 5.5, it turns out, just says hold all the time." As transcribed, it said hold on 41 of 45 calls. One answer repeated is perfectly consistent, so the check passed. Voss said that is probably worse, and that the model needs more work on being too conservative.
He argued that evals let a team move from guessing to showing that a change works. At the start of the session, he said, you cannot switch models without them.
Why it matters
Our reading: a score that rises is not proof that the product improved. A team that watched only the consistency number would have paid four times more for an agent that dodges the decision. That is how AI multiplies an old weakness. Teams already trust a headline metric, and now a model change can move it for the wrong reason, at a higher price.
For anyone who buys or runs AI features, the practical step is to ask for before-and-after results on every model change. Look at what the agent actually did, not only the score. As our own observation, a 13-minute eval run is modest next to committing to a model that costs four times as much.
The other side
Voss did not call the upgrade simply bad. Faithfulness improved a lot, and he judged the hold problem probably worse, not certainly worse.
Whether four times the cost is worth it, he said, each team must decide for itself in production. This was also one demo with one agent. It does not show how often upgrades fail this way.
Evals have limits too. Voss warned that every eval has a blind spot. The consistency check was fooled here, and other checks can be.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- AI Engineer: How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI (2026-10-17)
