- A demo banking agent cost under $7 for 1,400 interactions in a week, but the same dashboard showed unresolved disputes, 18% thumbs-down ratings and investment-advice risks.
- Customers carry the unresolved disputes and the firm carries the liability, so cost per interaction alone is an incomplete measure.
An agent's low running cost tells you little until you also count its failures. A workshop on the AI Engineer show, led by Langfuse's Doneyli De Jesus and Annabell Schäfer, put both on one dashboard for a banking assistant.
What was said
A speaker on the show, who said he once worked in a contact centre, showed a dashboard for a demo banking assistant. Over seven days it handled 1,400 interactions that needed no human. He put its cost at under $7. A human worker, he noted, earns $15 to $20 an hour and could not handle 1,400 calls in one hour. He called this a directional benefit and added that every organization is different.
The same dashboard listed failure modes, meaning the kinds of wrong result the team had defined in advance. Unresolved disputes ranked in the top three, and he said all three had been creeping up. In those cases the customer either reached a human or the session simply closed with nothing done.
He then pointed to the thumbs-down ratings and cited 18%. He also showed 89 cases in seven days where a customer asked for investment advice and the agent may have tried to give it. As a speaker on the show put it: "For financial services, for example, it should never give investment advice." Another warning came earlier: "AI agents can fail while the system metrics themselves look green."
Why it matters
Our reading: a buyer who sees $7 against $20 an hour is looking at one column. Other people pay for the second column. A customer with an unresolved dispute either gets passed to a human or is left with nothing. The firm carries the liability when a bank's agent strays into advice. The speaker named liability, financial and reputational effects as the stakes of customer-facing agents.
The cheap number is also the easy one. The failure counts in this demo exist only because the team defined failure modes and tracked them. Anyone buying or running an agent can ask for failure counts next to the cost per interaction.
The other side
The speaker never claimed that $7 was the full cost. He offered it as a first sign and moved straight on to the failures. The excerpt does not say what the $7 covers beyond running the agent.
The figures also come from a demo, not a real bank. Flagged cases are not proven harm: in the 89 investment cases, the agent only may have given advice. The talk gave no failure rate for human staff, so it cannot say whether people would do better.
The speakers' answer is better measurement. Filtering by failure mode let them cut thousands of traces down to dozens or hundreds worth investigating.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- AI Engineer: Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse (2026-10-18)
