- A Shandong University preprint found that, with the model fixed, coding-agent harnesses changed cost per task by up to 3x. Two very different harnesses also scored within five points on 447 tasks.
- Small claimed gains from a single run can be noise, so a vendor's accuracy pitch may not justify a higher bill.
- Ask vendors for cost per solved task, repeated runs of both setups, and preamble size, then test on your own tasks more than once.
What a harness actually sells
When a company buys an AI coding agent, it buys two things: a language model and the software around it. That software is called the harness. It holds the system prompt (the standing instructions), the tools the model can use, and the rules for what the model remembers.
Vendors often advertise accuracy gains from harness updates. A preprint from Shandong University researchers tests that idea. Its title gives the answer away: "What Does a Harness Buy? Tokens, Mostly." Tokens are the units AI providers bill by.
The lesson here is that harness choice shows up in the bill more clearly than in the score.
Similar scores, very different bills
The researchers tested three harnesses that the paper calls production tools: Claude Code, mini-SWE-agent and OpenCode. Each was run with five different models on SWE-bench Verified, a set of real software repair tasks. To learn how much a score drifts by chance, the team also reran identical setups.
On 447 tasks, with two models, the heaviest and lightest harnesses were equivalent within five points. Those were Claude Code and mini-SWE-agent. VentureBeat describes mini-SWE-agent as about 100 lines of Python with one bash tool and no context compaction.
The one harness effect that cleared the noise was a loss. On the larger task pool, OpenCode fell behind by as many as 9 points. For one model, the researchers say half of that shortfall came from runs where its output cap cut the work short.
Cost told a different story. Holding the model, the tasks and the price list constant, the researchers found that cost per task differed by up to three times across harnesses. The summary does not say which two harnesses sit at the two ends of that gap.
How the bill gets built
The researchers trace the gap to the first call. Every harness sends a preamble with each step: the system prompt plus tool schemas, which are written descriptions of each tool. A longer preamble is paid for again at every step.
The number of steps then multiplies that cost. Other drivers, such as how fast each step's context grew and how much output each tool call returned, varied much less between harnesses, the researchers report. The provider's price for cached input scaled the bill but did not change which harness cost more.
Think of a consultant who rereads a long brief aloud at the start of every meeting and bills by the minute. The advice may be no better. The invoice is larger.
Why one benchmark run misleads
On a 45-task hard subset with five models, swapping the harness flipped 13% of tasks. Rerunning the same harness also flipped 13%. The tasks a harness won in one run were not the tasks it won in the next.
That limits what small tests can show. Using the run-to-run swings they measured, the researchers calculate that a 45-task test would detect a 13-point difference in only half of attempts. A 447-task test can separate differences of about 5 points. They say that is still too blunt for many of the improvements harness changes claim.
VentureBeat adds context from the paper. As one example of the claims at stake, it notes that Zhipu AI says its ZCode harness beats Claude Code by 2.4 points when both use the same model. The paper also cites another study in which ten repeats of one unchanged setup on SWE-bench Verified produced scores spread across 2.2 to 6.0 points. Without a rerun of the baseline, a 2.4-point claim is hard to read.
Test conditions matter too. The researchers audited early runs that had web access. They found that 312 of 439 trials with DeepSeek and Qwen models, or 71%, had pulled the original human-written fix from GitHub. Those runs were thrown out, and the final tests ran with the network cut off.
Limits of the finding
This is a preprint, and it covers one benchmark. The 447-task result used two Qwen models only. The study does not test your codebase. It shows how hard it is to read a harness claim, not that every heavy harness is wasteful.
What to ask your team
Ask for cost per solved task next to pass rate. A tool that scores about the same as a cheaper one but costs up to three times as much needs a reason.
Ask whether a vendor's claimed gain comes from one run or from repeated runs of both setups. Without a rerun of the baseline, a gap may be noise.
Ask how large each harness's preamble is and how many steps it takes per task. Those two numbers drive the bill, according to the researchers.
Run any pilot on your own tasks, more than once. A harness is a meter before it is a muscle, and only repeated measurement shows which one you are paying for.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Shandong University researchers (arXiv preprint 2610.04433).
