Assume a tricked AI agent is fooled, and enforce limits outside the model

Emil Chetty leads data and AI product at Wise, the international payments company. On the AI Engineer show he argued that a company should assume a prompt-injected agent has been fooled. A prompt injection is hidden text that tricks an AI into following an attacker's instructions. His answer is to put the limits in ordinary software outside the model. That software can refuse a payment even when the agent believes the trick.

What was said

Chetty described how Wise lets an agent act for a customer. The customer confirms the whole job once, after seeing the outcome. That confirmed job becomes a fixed record the agent cannot edit. In his words: "We need the customer to agree to the job, the agent to choose the steps, and our software to enforce that agreement."

He said code then checks three things: the confirmed job, what Wise permits the agent to do, and the customer's current rights. Only the overlap becomes limited access for the workers that make the payment.

Then he gave an attack. Suppose the agent is prompt-injected somewhere and told a payee's details have changed. Suppose the agent believes it. Chetty said that belief cannot change the confirmed job, so the new payee fails the service check. "So the boundary doesn't actually depend on the model spotting the scam." He added that Wise relies on the payment software it has built since 2011.

Why it matters

Our reading: this is a question buyers and builders can ask of any agent product. Where is each limit enforced? If a rule lives in a prompt or in the model's judgement, a successful trick defeats the rule as well. If it lives in a separate service that checks every action, the trick has nothing to defeat.

It also changes what existing systems are worth. Chetty's design leans on payment services Wise already trusts and audits. Teams with strict, boring back-end checks may be better placed to use agents than teams that plan to rely on a smarter model.

Chetty also described a cost to the old approach. Asking customers to approve each step, he said, "We ultimately made the human the API." One agreed job spares them that queue.

The other side

Chetty did not say the model layer can be dropped. He said Wise still needs both the evals, which are tests of agent behaviour, and the harness around the agent. The boundary works only if downstream services can check and enforce the agreement. That suits a firm with a mature payments stack. A company without one has little to enforce with.

The design also does not remove every interruption. If a quote changes or expires, Chetty said the customer must approve the new version. The agent can prepare it but cannot approve it.

The excerpts also leave open how a customer judges the proposed job in the first place. A badly framed job, once confirmed, would be enforced just as faithfully. Chetty gave no test results or attack figures, so the claim rests on his description of the design.