- Evan Kormos of Coinbase described agents moving from full human review to sampled review to exceptions only, with a policy threshold triggering each move.
- The threshold sets when human coverage shrinks. The excerpts do not say how samples are drawn or exceptions found, and those rules decide what goes unseen.
Agents can earn less human oversight in stages, and a policy threshold marks each step back. Evan Kormos, a player-coach at Coinbase, described this path on the AI Engineer show. The threshold decides when human coverage shrinks. It is a number someone chose.
What was said
Kormos described work as a chain of steps, drawn as a graph. Inside each step, an agent runs a loop: it tries something, checks the result and tries again if needed. He said, "The loop is really only as trustworthy as the check."
He then laid out three stages. On day one, the process is supervised. People verify the agent's output, refine the data sets and define good work with subject experts.
When measured confidence reaches a policy threshold, the process moves to sampling. Humans read a slice of the output, not all of it. Systematic evaluation of the agent's traces, its recorded steps, is meant to show that the unseen part still meets the quality bar.
Once a further confidence threshold is cleared, humans handle more of the exceptions. Kormos said the decision step, the check against policy, should probably stay human-driven. He also said confidence should rest on evidence. In his words, "So, it's not a feeling about a model."
Why it matters
In the excerpts, Kormos does not say who bears the cost of mistakes. That is our reading. Each stage reduces human coverage, and a number sets the timing.
What goes unseen then depends on two things: how the sample is drawn and how exceptions are detected. An error outside the slice may reach no reviewer. It could land on whoever receives the output first, such as a customer or a small business downstream.
Leaders who build or buy agent workflows can ask direct questions. Who set the threshold, and on what evidence? How is the sample chosen? What counts as an exception, and who decides? These choices shape the cost of failure as much as the model does.
The other side
Kormos argued for measurement over gut feeling, and he kept humans on the policy decision. He also described people staying involved in refining the data sets after the move to exceptions.
The excerpts leave real gaps. They give no threshold values and no sample sizes. They do not say how an exception is detected, or whether the agent flags its own doubt. They also do not say what happens when a miss gets through.
Kormos did say, "The agents are probabilistic, and so the same request can produce different answers." Probabilistic means outputs vary. That makes any sample harder to trust. In the excerpts, no one answers who bears the loss when sampling misses something.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- AI Engineer: How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase (2026-10-19)
