Limiting what an AI may request turns its risk into an engineering problem

Karthic Rao of Perpetual argued on the AI Engineer show that AI in finance and insurance workflows is safest when the model's choices are narrowed. His team does not rely on the model to behave. It gives the model a closed list of allowed actions, checks every plan against that list, and ties each human approval to the exact thing being sent.

What was said

In Rao's design, a planner model only translates a request into a structured graph of intents, entities and facts. A compiler then checks that graph. The compiler never calls a model. It runs 11 deterministic stages and either produces an execution plan or fails closed. Two of its three verdicts are "needs clarification" and "refused". If a request names an ambiguous client, the answer comes back as a question, not a guess.

The plan is checked against a grammar: 42 typed action contracts covering forms, documents, email, e-signature, calendar and human steps. An action outside the list, or wired up wrongly, is refused. Rao said there are about 40 ways to fail closed. The same contract definition is pinned in the engine, the back end and the interface. His summary: "The model can only ask for things the firm already knows how to do safely."

Approval works the same way. A person approves the plan, and each external action, such as an email or signature request, has its own gate. That approval is tied to a hash of the exact payload and checked again just before sending. Once a plan is approved, a chat message cannot change it. There are no automatic retries. Unclear outcomes go to manual review. Rao put it this way: "The model steers. It never drives."

Why it matters

For buyers and managers, Rao's design suggests three questions for any vendor. What actions can the model ask for? What happens when it asks for something unknown? Is a human approval bound to the exact message or only to a general intent? The grammar also adds steps the planner never requested, such as rendering the PDF, because the contracts require them. Safety work is built into the structure, not left to the prompt.

The other side

A closed list only works where a firm can list what it does safely. Rao's example has 42 contracts in one domain. The excerpts do not say how that scales to messier work.

The model still makes judgments inside the boundary. A reply judge can mark a client's answer as partial, but that call is provisional until a person confirms it. The excerpts do not say how often approvers catch errors, or how to stop them approving out of habit. Dropping automatic retries also pushes unclear outcomes onto people.

Rao's cost claim is a design target. He said they designed for about 95% of model decisions to need only a small, fast model, and showed an illustrative run of 20 decisions. He said logging and evaluation exist today, but the training loop that would shrink the share of hard cases is next.