A Langfuse engineer says a calculator lifted a bank AI's number accuracy to about 95%

Doneyli De Jesus of Langfuse described a case where handing a weak task to ordinary code lifted an AI model's accuracy. On the AI Engineer show, he said a calculator tool took a small model's accuracy on numbers at a bank from 60% to about 95%.

What was said

De Jesus said his team helped a top-five bank test several models, small and large. One small model scored 97% on groundedness. That means it based its answers well on the bank's own internal data.

But it kept failing when an answer needed financial figures or simple sums. He explained that language models are weak at maths because they only predict the next word.

Instead of replacing the model, the team gave it a tool for the task it kept failing. The tool is a small piece of Python code the model calls whenever a number must be worked out. His reasoning: "Because math is deterministic." A calculator gives a fixed answer every time.

He said accuracy on numbers rose to about 95%. With an evaluation framework added on top, the model cleared the bank's 85% gate for this use case. He said banks approve applications, not models: "you have to gate the application, not the model."

Why it matters

Our reading: a strong headline score can hide one weak spot. The 97% groundedness figure alone would have looked like a pass.

De Jesus made a similar point about averages. He said a high average across qualities does not settle it. Each quality gets its own threshold, and risk experts help set the bar.

For anyone buying or running AI, the lesson is practical. Find where the model is weak and test that spot on its own. Then ask whether code that answers the same way every time can take over that task.

De Jesus also said this approach keeps an application's quality steady if the model is swapped later.

The other side

This is one bank, one use case and one account. It comes from someone at a company that builds evaluation tools. The conversation did not test the claim elsewhere.

The result was also not perfect. He gave the figure as about 95%, and later as over 90%. The excerpts do not say what the remaining errors look like. They also do not say whether that level is acceptable for every financial task, or how the larger models did on numbers.

A calculator only helps where code can do the job. For softer qualities, such as how complete an answer is, teams still use a language model as a judge. De Jesus said that "relying 100% just on LLM judges is not going to cut it." His answer was a blend of code-based and model-based checks.