- In a study of 1,368 AI incidents, weak human review and monitoring were more closely tied to harm than technical failure of the model.
- The researchers argued that AI should be tested as a whole system, with the most effort on the use cases where the stakes are highest.
Vishal Srivastava and Tanmay Sah of New York Life argued that weak human review and monitoring are more closely tied to AI harm than flaws in the model itself. They made the case on the AI Engineer show. Their work suggests that tests which stop at model accuracy cover only part of the risk.
What was said
The pair studied 1,368 incidents from a public AI incident database. On average, each incident involved six separate kinds of failure. They asked which kinds were most closely linked to harm.
Their chart put two factors near the top for harm. One was how well a human reviews the output. The other was harm detection, meaning whether monitoring is in place. Scale and system design showed up often, but were less tied to harm.
Grouped together, deployment controls came first, then business impact, then technical failure. A speaker on the show put it plainly: "all these sub-dimensions are important, but deployment controls are far more important."
The second speaker split risk into four parts: how often the model fails, whether the failure reaches a real decision, whether it goes uncaught, and how severe it is. He added that "the same model can create different risk depending on how we deploy it". They applied their testing method to an airline chatbot that gave a customer the wrong refund policy.
Why it matters
Our reading: a model score tells you little about the setup around it. Buyers and builders should ask who can read the output before it reaches a customer. They should also ask what catches a bad answer afterward, and how much authority the system has.
The speakers also suggested ranking use cases by how much is at stake. They argued that the riskiest ones deserve the most testing effort. A wrong refund answer reaches a customer first, so that is where review and monitoring should be strongest.
The other side
The finding is about association, not proof of cause. The speakers said all the sub-dimensions matter, and technical failure still ranked in their grouping. The excerpts give no figures for how strong the link is.
The data comes from reported incidents in a public database. The excerpts do not say how those incidents were gathered, so some failures may be missing.
The researchers also warned that review tools need checking too. They said the evaluator is part of the system that must be governed. The excerpts end before they explain how to test agent-style systems.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- AI Engineer: 1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life (2026-10-22)
