Coinbase's Evan Kormos: in regulated AI work, defining 'good' is the hard part

Evan Kormos says the hard part of putting AI agents into regulated work is defining what a good result looks like and building a check that tests for it. Kormos has spent 15 years in big-company financial services and now works as a hands-on team lead at Coinbase. He spoke on the AI Engineer show.

What was said

Kormos described a process as a chain of steps. Inside each step, an agent works in a loop: it does the work, then a check decides whether the result is good enough to move on. He called defining a good result the hard part. Without a dependable way to judge the output, he said, extra passes do not necessarily improve it. His summary: "The loop is really only as trustworthy as the check."

He contrasted this with traditional software, where unit tests, which are automated checks on code, build confidence. Agents are probabilistic, he said, meaning their answers can vary from one try to the next, and several different answers may be acceptable. Matching one exact output is therefore not enough. He called for test sets that reflect real cases, repeated evaluations run in large numbers, and traces, which are records of what an agent did while running.

Kormos used a lending example: identity checks and financials in parallel, then a credit memo, risk review, documentation and a decision. He suggested the decision stay with humans, since it is the check against policy. For rollout, he described three stages. On day one, humans verify all output, and subject matter experts help set the standard. Once measured confidence passes a limit set by policy, humans review a slice. Later, humans handle mostly exceptions while still helping improve the test data.

Why it matters

The other side

Kormos expected the decision step to be harder to improve gradually, and he kept it with humans. The excerpts do not show how a team sets the policy limit or knows its test data is representative. They give no results or numbers from real deployments. They also do not show whether people who can define good are hard to find. Under sampling, humans never see most outputs, so the approach leans on traces to catch problems. Nothing said addresses who checks the check itself. This is one practitioner's framework, not evidence that it works at scale.