A regulated AI agent needs evaluation that keeps running after launch

A regulated AI agent cannot be checked once at launch and then left alone. Shiv Indap, who heads Upstart's voice AI effort, and Brooke Hopkins, who founded the testing firm Coval, argued this on the AI Engineer show. Upstart's agent takes calls from loan applicants who need to verify details. The guests described scoring every live call and re-testing every fix before it reaches customers.

What was said

Upstart decides about 90% of applications instantly. The rest phone in to confirm details, and the agent asks three set questions. The rules are strict. If an applicant says the money is for medical use, a speaker said, the agent must not ask the amount.

On the stakes, a speaker said a slip is "not just a customer experience issue". It can also become a regulatory problem, and the speaker called that the worse outcome.

At launch, a human reviewed every call, first in a shared Google Sheet that soon sprawled. A second model, called an LLM judge, scored each finished call. At first the two disagreed often. A speaker said "a human reviewer and an LLM agent or an LLM judge agreed only 33% of the time." The show notes say agreement later reached 92%.

A speaker also said expert know-how had not been built into the agent at launch. Later, the guests described a loop. Score every live call. Send failures to human review. Replay simulated calls to test each fix. A speaker said relaunching a fix into production to see if it worked puts users at risk. Humans still approve every change.

Why it matters

Our reading: a release gate that closes at launch leaves a gap. The standard the agent must meet lived in reviewers' heads, and the 33% figure shows how far the first judge sat from it. A new model or a reworded prompt changes behaviour again. Without a standing test set and a way to re-run it, every change is a fresh gamble.

The cost of that gamble lands on the deployer. The speakers named a regulatory problem for Upstart. They also named risk to users. Managers buying agents could ask vendors how fixes are tested, and who signs off.

The other side

Coval sells evaluation tooling, and Indap said it helped Upstart build its human review process. So the case for constant evaluation comes partly from a seller of it. The excerpts give no figures for what the work costs.

The guests also conceded limits. A speaker said that at large scale, no system gets every case right. Humans cannot review thousands of calls a day, so the judge picks which calls need a person. The 92% agreement figure comes from the show notes, and the excerpts do not show how Upstart got there. Talk of self-healing agents was framed as promising, not proven.