- AssemblyAI's Matt Lawler said a bought bot solved about 10% of conversations and his team had no access to its prompt, tools or retrieval.
- He said the team's own agent resolved 80% in its first week for about $700 a month, but other differences may explain the gap.
AssemblyAI says its own support agent resolved 80% of tickets in its first week, after a bought bot solved about 10%. Matt Lawler, forward deployed engineer lead there, told the AI Engineer show that the bought bot was a black box. He named two problems: too few tickets solved, and no way to fix the bot fast. He tied the second to not having access to its prompt, tools and retrieval.
What was said
Lawler said his team was drowning in requests. They bought a ready-made chatbot and pointed it at their documentation. It solved about 10% of conversations. At a thousand a day, a hundred closed and 900 tickets still reached the team.
The second problem was speed of improvement. The team could not fix the system fast enough. In his words: "We did not have access to the system prompt." The same went for tools and for retrieval, the step that finds the right documents for each question. Each time the team asked for a change, the vendor said it was in the development plans.
The replacement, Joey, uses the Claude Agent SDK as its base. It keeps its own files. The team syncs its docs as local markdown files whenever they change. Voyage embeddings help it find the best-matching documents. If Joey is confused, it searches those files itself. A large instruction file, CLAUDE.md, shapes its behaviour, and Lawler said the team updates it after conversations. Joey also remembers earlier chats and works inside the support system, Pylon. Because the team owns all of this, a fix is a code change. Lawler said a new version goes live in about 30 seconds on Railway.
His result: "We went from a 10% to an 80% end-to-end issue resolution rate in just the first week of deploying this build". He put the cost at about $700 a month, including tokens and infrastructure. He said the 80% covers all incoming requests, not a chosen subset.
Why it matters
Our reading: before buying an AI support tool, ask which of the prompt, tools and retrieval you can see and change. Ask how fast a fix can ship. With a closed product, every fix is a vendor request. With an owned one, it is a code change. Lawler's account suggests the price of a closed bot is waiting on someone else's roadmap.
The other side
Control is Lawler's explanation, not a demonstrated cause. Joey differs from the bought bot in many ways: it is an agent, it searches files, it uses embeddings, it has memory and it has a large instruction file. The jump to 80% also came in the first week, so the fast-fix loop cannot explain it alone.
The figures are the company's own, from a talk. He did not say what the bought bot cost, so nothing here shows buying cost more in money. The $700 excludes the time to build Joey. AssemblyAI employs forward deployed engineers and sells AI products, so it may be unusually well placed to build. The bought bot was one unnamed product. Joey still passes about 20% of requests to humans.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- AI Engineer: We Built an AI Support Agent That Resolves 80% of Tickets — AssemblyAI (2026-10-04)
