- Fuad Ali of Arize AI argued that a voice agent's text log can look fine while the call itself failed. His example refunded the wrong order.
- He said teams need audio, transcript and trace together, plus checks that run on the audio. Customers bear the cost when text-only audits miss failures.
A voice agent's written transcript can show a tidy self-correction while the call itself went wrong. Fuad Ali, a senior product manager at Arize AI, made this case on the AI Engineer show. Teams that audit voice agents from text logs, he argued, are examining the wrong record. The caller is the one who pays when that record misses a failure.
What was said
Ali played a sample refund call. In the text log, a caller asks for a refund on order 14. The agent replies that it has begun one for order 40. The caller objects, and the log reads as though the agent fixed its slip.
The recording showed otherwise. Ali listed a 2.4-second silence before the agent spoke, the agent cutting across the caller, a misheard order number and a flat, robotic voice. The tool call, which is the step where the agent acts on the order system, showed the outcome. Ali said, "the refund was processed for order 40 and not 14, which is a huge, huge mistake."
A person reading only the text would miss all of it. In his words: "If you just read this from an LLM output, you would have no idea anything is going on." His fix is to review the recording, the transcript and the trace (the step-by-step record of what the system did) together in one session. He also wants automated checks that run on the audio itself, such as tone, interruptions and the delay before the first sound.
Why it matters
Our reading: a review of text can pass a call that the customer experienced as a failure. If your quality process samples transcripts, its pass rate may flatter how calls actually went.
The cost lands on the caller first. In Ali's example, someone gets a refund on the wrong order. Someone else must then notice and reverse it. Ali named customer support and drive-through ordering among the places voice agents are used.
For anyone buying or running these systems, the practical test is simple. Can a reviewer hear the call and see the action the agent took, side by side? If the tool shows only text, ask what it cannot see.
The other side
Ali works for Arize, which sells tooling for exactly this problem. He did concede that teams could build it themselves. He said it gets hard at very large scale.
The refund call was a demonstration, played as an AI-generated example. He gave no figures on how often such failures happen in real deployments.
One gap remains. The wrong order showed up in the tool call, which is a structured record rather than a transcript. The talk did not say whether a team that logs tool calls alongside text would catch that error. It also did not say what audio review costs, or how many calls need it.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- AI Engineer: The Transcript Looked Fine. The Call Wasn't. — Debugging Voice Agents, Arize (2026-10-05)
