In a crafted demo, an AI summary dropped a dual-approval line and money moved

Guests on the OWASP Foundation podcast argued that an AI agent's summary of a long conversation is a security weak point. Rico Komenda and Barno Kaharova discussed the risk. In a crafted demo, a dual-approval line was lost in summarisation, and a finance agent still moved money.

What was said

The guests called it the persistence trap, a fancy name for context or summary poisoning. Long agent sessions near the limit of what a model can hold, so the history gets summarised. A speaker said call and meeting transcripts are often summarised by language models too.

In the demo, a crafted conversation said the company needs dual approval. (Our gloss: two people must sign off a payment.) The conversation was compressed. Then a speaker asked ARIA, a demo finance agent, to act. The speaker said the line was lost in summarisation, guessing it was because "the context was full, or we wanted to have less tokens". Before the run, the speaker was unsure the attack would work.

The result, in the speaker's words: "the wire still got transferred". The speaker added that someone in the conversation had said maybe they need to do this. The agent acted on its memory and did not verify it against a deterministic rule set.

The guests said the demo is built on conversational patterns. They said repeated material tends to stay in a summary. They said material told as a story gets more weight. They said hedged wording, such as maybe, tends to get lost. One line from the talk: "AI agents are only as trustworthy as what they remember."

Why it matters

Our reading: a summary is written by a model that must fit the history into less space. It chooses what to keep. A rule that exists only as a line in that history is one more line it may leave out.

Our view is that buyers of payment-touching agents can ask three questions. Do approval rules live in code outside the model's memory, or only in the conversation? Is every payment checked against a fixed rule set before money moves? Can we see what a summary left out?

The other side

This was a demo with a conversation built to make the attack work. It does not show how often real summaries drop approval steps. The guests gave no failure rate for real finance deployments.

The guests also said this kind of attack is hard to defend. Separately, a speaker said responsibility is most times shifted to the developer, calling it shared responsibility all over again. That remark was not tied to this demo. Who bears the cost when an agent moves money after losing a control was left open.