- An AI agent can write a well-formatted, confident report that is wrong, and format checks will pass it.
- Laurie Voss argued that checks of meaning and a close read of the agent's step-by-step records help catch these failures.
Laurie Voss, who runs developer relations at Arize AI and helped start npm, argued on the AI Engineer show that confident wrongness is among the biggest risks with AI agents. A polished report can pass every format check and still be wrong. Checks that test meaning, plus a read of the agent's traces, are what catch it.
What was said
Voss built a financial analyst agent on Claude for the workshop. Agents chain many steps together, Voss explained. An early mistake can therefore cascade, pushing each later step further off course.
To show the type of failure, Voss offered a hypothetical. Ask such an agent for a report on Tesla, and the search returns results about the 18th century inventor instead of the car company. The agent writes a polished report on the wrong subject. No check fails, because the format, tone and grammar are all fine.
The real failure came from the test runs. One Tesla report gave analyst opinions that added up to 116%. The agent had run two web searches and merged figures from different data sets. Each search result was fine. The fault, Voss said, was in the reasoning.
Voss summed it up: "Confidently wrong is worse than obviously wrong." Voss urged teams to look for the cause, not just the wrong answer. Traces, which are step-by-step logs, show whether the fault was bad retrieval, bad reasoning, invented numbers or a question the agent misread. Each cause needs a different fix.
Why it matters
Our reading: many teams judge agent output by how it looks. Does it follow the template? Does it sound professional? The 116% report shows that a result can look finished and still rest on bad math. A manager approving an agent for customer or investor work should ask what tests check meaning, not just shape.
Voss said code checks catch formatting problems but miss problems with meaning. AI judges, meaning models that grade other models' output, catch reasoning gaps but miss some subtle made-up facts. Voss argued for stacking code checks, AI judges and human review, so an error must slip past every layer. For buyers, a fair question for any vendor is which layers they run, and whether you can see the traces.
The other side
Voss did not claim a perfect fix. "Every eval you have is going to have a blind spot." Stacked checks make errors harder to slip through; they do not remove the risk. Humans are still needed to check that the AI judges are judging correctly.
Tests can also be wrong. Voss described an Anthropic benchmark in which Claude found an unexpected way to meet a customer's flight request, and the test marked it as a failure. A failed test does not always mean a bad agent.
Finally, only the 116% case was a demonstrated run. The inventor example was a hypothetical, and the excerpts give no figure for how often agents fail this way in real use.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- AI Engineer: How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI (2026-10-17)
