The cheaper AI model can cost more per task and score lower

Rayan Krishnan of Vals AI argued on the AI Engineer show that price per token is a poor guide to what AI costs. A model that looks cheaper on a price list can cost more per finished task and do worse work. Firms that do not measure cost per task against performance, he said, risk overpaying.

What was said

Vals AI builds benchmarks, which are standard tests that score AI models. Krishnan pointed to a tax benchmark. He said Anthropic's second-tier model, which he called Opus 5.5, is cheaper on paper than the frontier model, which he called Fable 5.1.

In practice it cost more. The less capable model used more tokens, the units of text AI vendors bill for, to get its work done. It also scored about 3% lower. His conclusion: "You would have spent less and got a better performance."

He added that the best model changes by task. On Excel modelling for leveraged buyouts, he said the top general model dropped off the cost-performance frontier.

He then described his own team. In one month of free, unlimited tokens, "we spent $1.5 million worth of tokens." He thought the team was 10 or 12 people, and he put the bill at roughly ten times their salary cost.

To cut that, they built evals, meaning tests drawn from their own work. They took past GitHub pull requests and gave the same tasks to agents. That showed which models were overkill. Krishnan said he thinks spend is now about 90% lower for the same output.

Why it matters

Our reading: the vendor's price list is not the number that matters. Whoever signs for the tools pays for any gap between list price and real cost per task, and pays again if the work is worse.

Krishnan said the same tests can guide which tools a company buys, what usage limits to set, and which model suits which task. That is a procurement habit, and it needs a record of past work to test against.

The other side

Krishnan's company sells this kind of testing. He said his team turned its internal tools into a product. That does not make him wrong, but readers should weigh it.

The 90% figure was his own estimate, hedged with "I think". It also started from an unusual high: a month of unlimited tokens with no limits. Many firms would begin far lower and save less.

The tax result came from one benchmark. Krishnan himself said rankings shift by task, so no result carries over without testing. He also conceded that most benchmarks use a single prompt and reply, while real work is multi-turn. Evals have to cover that, and he did not say how far that is solved.