- Laurie Voss said an AI agent's answers vary from run to run, so one success proves little. Two scores, pass at k and pass to the k, measure different things.
- For a finance analyst or a support bot, he argued, the agent must pass every time. For a coding helper, a correct answer after one retry is acceptable.
An AI agent that gets it right once has not shown it is safe to use. Laurie Voss, who leads developer relations at Arize AI and co-founded npm, argued this on the AI Engineer show. For finance and customer support, he said, the test is whether the agent is right every time it runs.
What was said
Voss said agents are non-deterministic, meaning the same input can produce different answers on different runs. So he ran his tests three times. Any single run, he said, could point him the wrong way.
He described two ways to score repeated runs. Pass at k counts a case as won if any one of k attempts works. Pass to the k counts it only if all k attempts work.
As k grows, the two scores pull apart. More attempts make pass at k look better, since one success is enough. They make pass to the k look worse, since one miss ruins it.
He matched each score to a job. For a coding helper, a correct answer after one retry is acceptable, so the looser score works. A support bot that errs regularly is a serious problem, he said. Of the finance case, he put it this way: "A financial analyst that gives inaccurate recommendations is also a huge problem."
His demo agent wrote a stock report saying 116% of analysts agreed on something. Voss traced that to faulty arithmetic. His summary: "Confidently wrong is worse than obviously wrong."
Why it matters
Our reading: a demo that works once is pass at k evidence. It shows the agent can succeed, not that it will every time.
In finance and support, the agent runs again and again, and each run is a fresh chance to fail. A small error rate per run adds up across volume. Customers and clients meet those failures, not the vendor.
The 116% report shows a related trap. It looked polished, and a single clean-looking output proves little. Voss did not tie this particular error to run-to-run variation, so it is not an example of a failure that repeated runs would expose.
For buyers, this suggests asking vendors for results across repeated runs of the same cases. For builders, it suggests running each case several times and counting it as passed only if every run passes.
The other side
Voss gave no rule for how many runs are enough. He ran his own tests three times.
He also warned that small test sets are noisy. To cut the margin of error in half, he said, a team needs four times the examples. Going from 25 to 100 is worthwhile, but 400 is where people stop bothering.
He conceded that "Every eval you have is going to have a blind spot."
His own fix was partial. A one-line prompt change lifted actionability, meaning whether the agent made a good judgment call, to perfect. Faithfulness, meaning whether it used only numbers from its research, went from 2 of 16 to 3 of 48. Voss said the agent was still inventing data some of the time.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- AI Engineer: How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI (2026-10-17)
