A benchmark maker says one model-card score is a weak guide to what to buy

Rayan Krishnan of Vals AI argued on the AI Engineer podcast that one benchmark score is a weak guide to which AI model to buy. Vals builds benchmarks that model makers cite in their launch material. He said the best model changes with the domain, the task, the programming language and the code repository.

What was said

Krishnan called this the jagged frontier: results inside one domain can differ from the single combined number. Even across Vals's own finance benchmarks, he said, it is not obvious which model comes first. Coding, he said, is more fragmented still. When Vals builds custom tests for one repository, the ranking splits further across languages and libraries.

His main example was Vals's Excel-modeling benchmark. Vals plots each model by cost per task against its score. Cost per task counts the whole job, not just the price of tokens going in and out. The best models form the edge of that chart, called the frontier. On it, no rival is both cheaper and better. For leveraged buyout (LBO) tasks, a narrower slice, the whole frontier sat lower than the overall score. One model that looked best overall dropped off the frontier there. Buyers, Krishnan said, would be "massively overpaying" for a model that no longer led that category.

He also said the accuracy metric changes the takeaway. Partial credit asks what share of the available points a model's work would earn on average. A fully-correct metric asks what share of questions it answers entirely right.

Why it matters

Our reading: a model-card number is an average, and a team does not run an average. Krishnan said a firm that mostly builds one type of Excel model needs a clearer picture of what is relevant to it. A buyer who picks on the headline figure may pay for a model that is not best at the work they actually do.

In our inference, that turns into questions for any vendor. Which tasks, languages and codebases produced this score? Was it partial credit or fully correct answers? What does one finished task cost, not one token? None of these questions was posed in the excerpts as a checklist.

Krishnan showed a chart in which 90% regularly use AI, while the smallest bar shows established ROI at 7%. The excerpts do not name the survey or say who was counted. He also described a team of "10 or 12" that spent $1.5 million on tokens. The show notes say it later cut spending about 90% with internal tests.

The other side

Vals builds the benchmarks that labs cite, so Krishnan has an interest in the idea that single scores need supplementing. The excerpts show detailed results only for finance and coding.

Krishnan described one comparison where an option scored 3% lower. Using the frontier model instead, he said, "You would have spent less and got a better performance." The excerpts do not say which models were compared, or whether 3% means points or a relative drop.

The idea that a model cheaper on paper can cost more comes from the show notes, not from data shown here. The excerpts also do not show how the $1.5 million team cut its spend, or whether quality held. The show notes list reward hacking, where a system games a test, as another lesson from building benchmarks. The excerpts leave open how costly good in-house tests are.