- Fuad Ali of Arize AI argued that agents should do the reading of huge amounts of data, while people validate fixes, curate datasets and design evals.
- He warned that an agent's findings are a starting point and must be confirmed as real before anyone acts on them.
Fuad Ali of Arize AI argued on the AI Engineer show that agents should do the heavy reading of agent logs. People, he said, belong at a few high-leverage checkpoints: confirming issues are real, setting what good looks like, and approving fixes. The further step, that judging becomes the engineer's main job, is WebPulse's extension of his talk, not his claim.
What was said
Ali described a loop built around "traces", the step-by-step records of what an agent did. He said nobody can read all of them. A chatbot serving millions of customers produces far too many. So an always-on agent called Signal reads the traces on a schedule and files issues. It then proposes a test, drafts a fix and runs an experiment to show the fix works. A person reviews the result and chooses whether to merge.
His demo ran a toy-shopping agent with a bug he had planted on purpose: it kept repeating product searches it did not need. Signal found it. It also caught a second problem in the trace records, which Ali said was just his own mistake in setting up the tracing.
On trust, his advice was general. Treat the agent's output as a starting point, validate the fixes and use your judgment on whether issues are real. In his words: "You can't just trust every single issue that comes out of this agent."
He named three places where people add the most value. One is reading the fix before it merges. Another is deciding which findings belong in the test set. The third is designing the tests, where, he said, you pass your taste of what is good on to the agent.
Why it matters
Our reading: this changes what a manager should look for in engineers. Reading logs and writing a first patch is the work Ali wants agents to take. The scarcer skill is knowing whether a reported problem is real and whether a fix is good enough. Ali put the theme this way: "Just because you can automate something doesn't mean it should be automated". Teams that buy such tools still have to decide who signs off, and on which changes.
Ali added one practical tool. Some models give a confidence score, and low-scoring cases can be sent to a person automatically. In his spam-detection example, he said, 4.6% of cases fell in a middle band and went to people, leaving 99.5% accuracy on the rest. He said the right threshold depends on the use case.
The other side
Ali works for a company that sells this kind of tool, and he said so himself. His demo was also a planted bug, which is an easy case. He conceded that "codifying qualitative decisions is very hard". Counting retries or measuring delay is simple. Judging whether an answer is good is not.
He also left open how much to automate. Merging fixes without review is, he said, up to each team, though he assumed high-stakes fields like finance would want a human to read every change. The talk did not answer how people build judgment if agents do the reading.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- AI Engineer: Agents That Self Improve While You Sleep - Fuad Ali, Arize AI (2026-10-26)
