Public AI benchmarks build a shortlist; only your own tests should decide

Public AI benchmark scores should help you build a shortlist. They should not make the buying decision. Stephanie Jarmak builds evals (structured tests of AI systems) at Omni Analytics. She made this point on the AI Engineer show. Her argument: a leaderboard rank says little about how an agent will behave in your own setup.

What was said

Jarmak once worked as an astronomer, and her time at Sourcegraph produced a large benchmark on coding tasks. She described a comparison from a third-party source, run on Terminal Bench, a widely used test for coding agents.

Two models were tried inside a harness. A harness is the software around a model that gives it tools and instructions. In one open harness, one model beat the other. Then the testers swapped only the harness, and the other model did much better.

Jarmak called that a difference of 30-plus percent in how well the configurations worked. Her takeaway was to evaluate the setup closest to your real work and environment.

Public benchmarks, she said, mainly help you understand the landscape and "get a short list of what you might be exploring". She warned against leaning on them too hard, because model training is uneven. A law firm, she said, should not read coding results unless it plans to give coding agents to developers.

For your own tests, she advised realistic tasks grouped by type and repeated runs, since these systems are not deterministic. She also advised keeping full traces. Her closing line: "ultimately, like, production is truth."

Why it matters

Our reading: a leaderboard number is a fact about someone else's setup, not yours. Managers who buy on it are trusting a figure that may not carry over.

In practice, this means a few changes. Ask vendors which harness produced a score. Ask for results on tasks like yours. Budget time to build a small internal test set from real work. Jarmak's example shows that the same model can rank differently in a different wrapper. Comparing models alone can mislead.

The other side

The 30-plus percent figure comes from one third-party comparison, which Jarmak cited loosely. She did not claim every harness swap moves results that much.

Her own advice also has a cost. Building realistic tasks, repeating runs and keeping traces takes effort. The excerpts do not say how many tasks are enough or what a team should spend.

Public benchmarks keep a role in her view, as a map of the field and a source of candidates. She also said she does not trust these systems "at all" even while automating. So a good internal test still does not settle the question. Production does.