Approve the whole AI application, not just the model behind it

Approve the application, not the model. Doneyli De Jesus, an AI engineering lead at Langfuse, a ClickHouse company, argued this on the AI Engineer show. He described work with a top-five bank. A model can look strong on one measure and still fail where errors cost money, so approval should cover the whole application.

What was said

De Jesus said the bank told his team directly: "we don't certify models here. We certify applications, right?" The bank's example was an analyst tool that works on earnings call transcripts.

His team first tested small and large models. One result stuck with him. A model scored 97% on groundedness, meaning its answers rested on the bank's own internal sources. He called that a passing bar.

The same model stumbled on accurate numbers, scoring 60%. De Jesus put it this way: "LLMs inherently are not great at math. They're just next token predictors."

The fix was not a bigger model. The team gave the agent a calculator tool for that part of the job. His reasoning: "Because math is deterministic. It's going to be the same all the time." The show notes say that with the calculator the system got past the bank's threshold of 85%.

He also warned against averages. Each dimension gets its own threshold. In his words, "it might be that one dimension actually completely shuts down the gate for you to move this into production."

Why it matters

Our reading: a single headline score hides where an application will hurt. A 97% grounded result sounds safe. Yet the same system scored 60% on numbers, and in finance a wrong figure is the costly error.

A budget signer can put three questions to any vendor. Which dimensions were scored, and what was the result on numbers? What threshold applied to each dimension, and who set it? Was each weak area fixed with a tool or a rule, or just averaged away?

De Jesus also argued that gating the application lets you swap the model later without losing the controls.

The other side

This is one speaker describing one bank's process, and the talk offers no outside evidence that it works elsewhere. The setup is heavy. De Jesus said such banks want on-premises, gated and air-gapped systems, which smaller firms may lack.

He conceded the field has no settled method: "there's no template for this," he said. Even a passed gate shows only that the app clears the tests someone chose. He also said relying on LLM judges alone "is not going to cut it."