Sending every AI alarm to a human reviewer turns oversight into a capacity problem

Stephanie Jarmak builds evals (tests of AI agents) at Omni Analytics. On the AI Engineer show she warned that when every automated alarm goes to a person, the review load can outgrow the experts available. In her example, a 1% false alarm rate would mean 10,000 reviews.

What was said

Jarmak was talking about quality gates, the checks that stop agent work. She said a gate should be enforced in the code itself, not just suggested. Some cases then get escalated to a human. That, she said, forces you to balance the human review load.

Her example came from auditing and repairing failures of an LLM as a judge, meaning one AI model grading another's output. The failures were in a separate system. That system saw a 1% false alarm rate. If every alarm it raised went to a person, she said, the result would be 10,000 reviews. She then asked whether an organization actually has the capacity for domain experts to do that work.

She said teams should think about where the bottleneck might be. They should also consider how to change the alarm rates, and how much each alarm affects what happens downstream. She is not anti-automation: "I want to automate as much as possible for the work to get done, but I also do not trust these systems, like, at all."

Why it matters

Our reading: a human in the loop is a staffing decision, not only a design choice. Sending every alarm to a person is not a safety plan unless someone has counted the people. If the queue grows past what reviewers can handle, one of two things can happen. Alarms pile up unread, or reviewers go through them faster than they should. Either way, the person named as the safeguard carries the risk of a miss.

For builders, this argues for deciding routine cases in code and sending people only the alarms that matter. For buyers, it suggests asking a vendor how many alarms a tool raises, who reviews them, and how many reviewers that takes.

The other side

The excerpt gives one example, from a different system, and does not say what happened to the reviewers. It offers no fix beyond changing alarm rates and weighing consequences. Whether a lower alarm rate also means more missed problems is not addressed.

Automated checks have limits too. In another example from the same talk, a hash check (a tamper test) passed everything, while an independent record comparison caught every injected security issue. Removing people does not remove the question of whether the checks themselves can be trusted.