- An eval is only as good as its answer key. Old reviewed audits can hold superseded results, bad evidence or outdated rules.
- Fieldguide's Anirudh Veeraragavan said such errors can punish a right agent or reward a wrong one, so a headline score tells you little.
A high eval score does not tell you whether your AI agent is good. Anirudh Veeraragavan, lead security engineer at Fieldguide, made this point on the AI Engineer show. The score can be wrong in either direction, he argued, because the answer key behind it may be wrong. His fix is to audit that answer key too.
What was said
Veeraragavan described an agent that helps with bank reconciliation audits. Fieldguide built its eval, a test that scores the agent's answers, on gold data. Gold data is the set of answers treated as correct. It came from last year's reviewed audits.
But he said a reviewed audit is not necessarily a correct one. Earlier conclusions may have been superseded since. Some decisions may rest on bad evidence. The audit method may have changed.
Those flaws cut both ways. In his words, mistakes in the data could punish a right agent or reward a wrong one. In his example the score was 97%. Some of the agent's misses were the eval's fault, so the agent was doing better than the score said.
Other cases ran the opposite way. The eval said no exception was needed. After business rule or audit method changes, that answer was now wrong. There the agent had really missed. He summed it up this way: "this kind of headline great overall score can hide a lot of underlying failures."
Why it matters
Our reading: a score describes the test, not only the agent. If you buy or run an agent on its benchmark number, ask where the answer key came from. Ask how old it is.
Ask also whether the rules have changed since the answers were marked correct. A vendor's 97% may be hiding failures. It may also be hiding strengths that a better key would show.
The audit trail matters most where a wrong answer has a price. Veeraragavan was speaking about audit work, where he said the conclusion must be defensible. He listed eval integrity as one of four controls and said you need all of them.
The other side
This is one example from one audit setting, told by a company that builds in it. Veeraragavan did not say how common flawed answer keys are. He did not say how much effort an audit of the data costs.
Re-checking old answers also needs experts who know how the rules have changed. That is the kind of skilled human time automation is meant to save.
The transcript's counts of misses are also hard to reconcile, so we have not repeated them. The audit finds that the score is unreliable. It does not say how to reach a single true number.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- AI Engineer: Your Financial Agent Needs to Show Its Work - Anirudh Veeraragavan, Fieldguide (2026-11-07)
