Choosing a frontier model was not enough to build Ramp's accounting agent

Ryan Stevens, who leads applied science at Ramp, said that picking a frontier model was not enough to build the company's bookkeeping agent. On the AI Engineer show, he described how Ramp spent four to six months building tests, called evals, for Stack, its agent for bookkeepers, before launch.

What was said

Stevens said the team broke a bookkeeper's job into tasks, such as reconciliation and variance analysis. Per the show notes, they then built synthetic businesses with full books, so the agent could be run against the same data day after day. He said four to six months of this is typical for agents in areas the big models are not fully tuned for.

The tests used two scores. Accuracy gives partial credit: four of six criteria right scores 67%. Pass@k is all or nothing. In his example, an agent got four, then five, then all six right across three runs, so it only passed on the third try. Stevens said accuracy on the latest models was very high while the first-try pass rate was very low.

Looking at why, his team found the agent usually gathered the right data and then reasoned about it wrongly. He summed it up: "the models, They just ain't that smart."

Why it matters

Our reading: the model is still a gap, and evals are how anyone finds out how big. A vendor can show a high accuracy figure while the agent often gets something wrong. Stevens named the stakes: "If an accountant can't trust that one of the steps is wrong, then they may not use your agent." A partial-credit score hides exactly the failures that decide that trust.

It is also judging work. Someone has to decide what "right" means for each task, build realistic fake businesses and grade the output. In Ramp's case, that slow work took months. That points to people who know the profession well enough to define and grade good work as central to an agent's quality. The excerpts do not say how such people are trained.

Buyers can turn this into questions. How long did you test, and on what data? Do you report all-or-nothing scores or only partial credit? Which tasks fail, and is the cause reasoning or missing data?

The other side

Stevens spoke about his own product. He did not call evals a moat, and he did not say they matter more than the model. Ramp also used its evals to pick the best frontier model, so they work alongside the model, not instead of it.

He said the tests are not saturated, meaning there is room to improve, and that changes to the agent's surrounding setup raised the all-or-nothing score. He also said "we've got some room to run here," which he framed as room for further improvement. Whether rivals can copy this testing work was not addressed. Stevens also reported that automatically tuning prompts did not work for his team, though he allowed it might be their agent.