At AssemblyAI, once an AI agent answers first, the human job becomes judging it

At AssemblyAI, once an AI agent answers every customer first, the human job shifts toward judging that agent. Matt Lawler, who leads forward deployed engineers there, described the setup on the AI Engineer show. People no longer handle first contact. They take the roughly 20% of cases the agent passes on, and they fix the agent after it fails. "Judging" is our reading of the work he described, not his own word.

What was said

AssemblyAI gets about 1,000 API signups a day. Lawler said his team first bought an off-the-shelf support bot, which closed about 10% of conversations. The team could not see or change its system prompt, its tools or its retrieval setup. Each change request went to the vendor.

So they built their own agent, Joey, on the Claude Agent SDK. Every chat, sales request or support email now meets Joey first. By Lawler's account, Joey closes four in five tickets with no human involved. He put the cost at about $700 a month, tokens and infrastructure included.

Humans still work around Joey. After a bad conversation, the team writes a fix and deploys it in about 30 seconds. Joey's rules live in a file called CLAUDE.md, which Lawler thought runs to about 30,000 lines. The team edits it after a bad customer experience or a wrong answer. In his words: "We are releasing updates for this Claude MD and now it gets better with each subsequent conversation."

Why it matters

Our reading: the support question moves from how many people answer tickets to who can review failures well. That person must spot a wrong answer, trace it to a missing or unclear rule, and write a better one.

For buyers, the 10% bot offers a second lesson. Without access to its prompt, tools or retrieval, the team could not run the fix-and-redeploy loop Lawler describes. Before buying an agent, ask whether your people can read its failures and edit its instructions.

The other side

The figures are Lawler's own, from a talk about his team's work. He said the 80% is not limited to a favourable subset of tickets. We cannot check that from the recording.

The talk also left open how people learn this judgment. If new staff never answer tickets, it is unclear how they build a sense for a bad answer. It also did not say how a file of roughly 30,000 lines stays consistent as edits pile up.