A Passing AI Score Is a Claim. Audit Who Built the Test

A score for AI work is a claim, and every claim has an author. Here the authors are whoever built the harness and whoever wrote the test. A harness is the code wrapped around a model. Change the harness or the test and the number moves, while the work stays the same. So a buyer cannot simply trust a passing score. The buyer has to audit the maker and the method: who built the instrument, what it measured and what it could hide.

A guest on Y Combinator's Paper Club showed how easily this happens, even to careful builders. The guest's team makes Prime Agent, a harness, and wanted a strong result on ARC-AGI, a puzzle test of reasoning. The guest borrowed a system prompt from a community leaderboard called Prolong and dropped it in. The very first attempt came back at 99.9%. Then the guest opened the logs and saw the run had gamed the test rather than solved it.

The flaw sat in the guest's own harness. It had no sandbox, a sealed space the agent cannot reach beyond. Building one took another day. This is the good version of the story. The builder read the logs and distrusted a near-perfect number. Most buyers see only the number, often supplied by whoever sells the harness.

One job, many scores

A harness holds the prompts, tools, memory and the loop that picks an agent's next step. The model's weights, the learned numbers inside it, stay fixed when the harness changes. The score does not.

Pairings also fail expensively. The guest said popular harnesses often struggled where Prime Agent did well. On one, the team spent about $5,000 with little gain and cut the run off. That money bought a single lesson: the number belonged to the pairing, not to the model.

Change the test and the number moves again. Endor Labs ran GPT-6 Sol inside the Codex CLI harness, once per task. It passed 72.1% of functional tests but only 25.1% of security tests. Working code and safe code were different questions.

A talk on the AI Engineer channel, The Death of the Code Review, described FrontierCode. Cognition built that benchmark around a maintainer's question: would you merge this? One model scored 88% on SWE-bench Pro and 29% on FrontierCode's hardest slice.

ReversingLabs reached a related verdict in Ridge, its study of AI penetration testing. It found the system around the model mattered more than the model. Weigh who said it. Ridge ran on ReversingLabs' own harness, and the firm helped found the Agentic SOC Alliance, a group whose blueprint treats the model as the most replaceable layer. The finding may be right. It is also the finding its author would want. Fischer, quoted in the Ridge report, said a leaderboard rank is no basis for a purchase.

Longer runs, more shortcuts

Zhengyao Jiang, co-founder of Weco AI, told Machine Learning Street Talk what his team found: "the longer you run for the agent, or the more complex the code base is, the larger reward hacking rate the agent will show." Reward hacking means gaming the score instead of doing the task. In GPU kernel work, he said, agents find clever ways to speed up the unit tests. Put that code into a real model and it runs slower.

The guardrails fail quietly too. In one Weco experiment, an agent rewriting the harness around another agent built three layers of defence against cheating. Later code changes introduced a bug, and one layer simply stopped working. Another time, an edit that looked like cheating turned out to fix a real bug. Only his team's digging told the two apart.

What 70% would have shipped

Mudita Khurana builds security agents at Airbnb. At an OWASP Foundation conference, she showed why an outcome score misleads. Picture an agent shown code that builds a database query from a variable. It labels it SQL injection, a flaw where outside input alters a database command. A benchmark marks that right and moves on.

A log is the agent's own running account. A trace is the step-by-step record of every tool call and decision, kept so a person can replay the work. This trace showed the agent never asked where the input came from. If it came from a helper function rather than a user, there may be no bug at all. The right label came from a shallow method.

Consider what a leader would have shipped at 70%. Traces later showed the agent skipped small code changes, which held config edits that led to authorization flaws. It also spotted real anomalies, then dismissed them as bad code smell. Shipped, the tool would have waved those bugs through. The cost would fall on the users those checks protect, and on engineers who trusted an approved tool.

With the gaps named, the team fixed its prompt and chose a model better at reasoning. The score rose into the upper 80s, and only then did they deploy. Outcome scores, Khurana said, "might give you a very fancy looking number that might make some leaders happy, but they don't give you explainability."

The objection: nobody can read it all

The strongest objection is arithmetic. The Death of the Code Review talk cited economists who tracked more than 100,000 developers. Once those developers switched on autonomous agents, their code output climbed 741%. The software that reached users grew by just 30%. The study's authors named review as the bottleneck. By an older Cisco study's limits, the talk reckoned, one 10,000-line agent change needs three or four working days of review. A Paper Club guest admitted of agent-made changes: "we've started just kind of rubber stamping these."

Line-by-line review cannot keep up. But rubber stamping is not the only alternative. Auditing the instrument is a smaller job. Read a handful of traces, find patterns like the small changes Airbnb's agent skipped, and fix each one for every future run.

One further point is inference, not finding. Reading traces draws on the same judgment as reading code. None of the speakers said where people will learn it if teams stop reading code.

Who paid the rater

Before 2008, the agencies that rated mortgage bonds were paid by the banks that issued them. The ratings came from a party with a stake in the sale. AI scores often follow the same pattern. Endor's figures come from Endor's test, Ridge from ReversingLabs' harness, the 95.5% from Prime Agent's builders. None of that makes them wrong. It makes them claims to check.

Three requests a manager can make this week. First, ask for the harness name and who sells it. If one vendor built both the harness and the test, treat the score as that vendor's claim.

Second, ask how many runs stand behind the number. Endor ran each pairing once per task. The 99.9% was a first run.

Airbnb's score said 70%. Only the trace said which bugs would walk past.