- In Langfuse's demo banking assistant, a presenter typed a card number and a Bitcoin question to show why controls must follow user input.
- The demo masked card data before storing it, scored investment-advice intent, and tested the assistant with red-team datasets.
A bank's chat assistant takes free text, so customers can type things its designers never planned for. Doneyli De Jesus and Annabell Schäfer of Langfuse argued that safeguards must be built for that. They made the case in a hands-on workshop on the AI Engineer show, using a demo banking assistant.
What was said
The bank in the workshop was a staged demo. In one test, the presenter typed a full card number, unprotected, to ask whether a lost card was blocked. The presenter flagged this as a problem. The saved trace, the step-by-step record of what the agent did, showed the number redacted.
A speaker explained the reason. Banks do not want customers' card or social security numbers sitting in an observability tool, which records how an agent behaves. They prefer to keep that data in their own core records. The speaker said "you cannot control what your users input in the system, you want to make sure that you catch that".
A second test asked whether to put savings in Bitcoin. The agent declined. An input guardrail scored the customer's intent as investment advice, so the team could filter for those cases.
The test sets followed the same logic. One worked as a red team. It probed for prompt injection, which means tricking an assistant into ignoring its rules. Teams can also add real production traces to a dataset. A speaker said "customers are always going to find very clever ways to use the system in a different way."
Why it matters
Our reading: a chat box turns anything a customer might type into data your systems must handle. Without masking, card numbers could end up in logs and monitoring tools. Buyers and builders can ask where typed input is stored and who can see it. They can ask what scores show the assistant stayed in bounds. They can also ask whether test sets come from real traffic or only from the designers' guesses.
The other side
This is a staged demo on the speakers' own product. The speakers call customers unpredictable, but the excerpts give no data on how often real customers paste card numbers. They also give no figures on how often masking or scoring misses. Masking covers only the data a team has defined as sensitive.
A speaker conceded that these models will sometimes give wrong answers. Another warned that dashboards can look fine even while an agent is failing. A speaker advised tuning checks to over-flag at first, so that nothing slips through. The excerpts do not say what those false alarms cost.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- AI Engineer: Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse (2026-10-18)
