- On Co-Design Bench, built from real coding cases that favored hard ones, the best of ten models passed 69.9% of criteria per task but fully satisfied only 26.4% of tasks.
- Partial credit hides the gap that matters. One unmet requirement can still send a change back to a person.
- Test the model and its working environment together, on your own past change requests, and measure whole tasks finished.
Seventy percent done is not done
A contractor who finishes 70 percent of your kitchen has not delivered a kitchen. The sink works and the cabinets hang, but the stove has no gas line. You still pay someone to come back.
A new coding test paints that picture for AI agents. Researchers from Alibaba and several Asian universities published Co-Design Bench on arXiv in October 2026. In their shared test setup, the strongest of ten models passed an average of 69.9% of the acceptance criteria per task. It fully satisfied only 26.4% of tasks.
That leaves 73.6% of tasks with at least one requirement unmet. The Decoder, which covered the paper, concludes that users must usually clean up by hand. That is the outlet's reading. The paper measures completion, not hours of cleanup.
What the test measures
The team drew on 26,314 real user coding sessions. From 37,463 distinct queries in them, it picked 273 tasks. Each task pairs a request with a runnable app and hidden criteria written by experts. There are 2,331 criteria in all.
The test has two tracks. One covers localized edits that may come with an image, audio, video or document. The other covers coordinated changes across a whole code repository.
The Decoder reports that the authors favored cases where users had to reword a request or roll back a change. So the test leans toward work that already gave agents trouble. It is not a sample of average work. The 26.4% is not a completion rate for everyday tasks.
How an answer earns credit
The grading protocol is called judge-then-refute. One AI judge checks each criterion using source code and browser evidence. A separate AI verifier then challenges every positive verdict. The paper says this happens without gold code or target screenshots.
The Decoder adds that the second check looks for mock-ups and hardcoded results. Points go only to features that work in the running app. A screen that merely looks right earns nothing.
Where agents fall short
Tasks that mix several kinds of change were harder. In the repository track, every one of the ten models did worse on edits touching three or four change types than on edits touching one or two.
The Decoder lists layout, design, interaction and content as the types, and says real projects tend to mix them.
The Decoder gives a small example of the work: take a figure from a receipt, then update every total that depends on it. Miss one total and the numbers disagree. That is how a task can score well on points and still fail as a whole.
More computing power did not close the gap. On smaller changes, the Decoder says GLM-5.2 consumed close to six times the tokens of GPT-6 Astra and still scored clearly lower. Tokens are the units of text a model reads and writes, and usage is typically what gets billed.
The model and its workspace come as a pair
The researchers also ran the same models in different harnesses. A harness is the program through which an agent reads, writes and runs code. The Decoder reports that Qwen3.8-27B scored 75% or 56% on larger overhauls, depending on the environment. The environment could also change which of two models placed higher.
The authors recommend reporting model and environment together. A leaderboard rank for a model alone does not tell you what you will get in your own setup. So the 26.4% applies to one shared harness, not to every way of running the model.
The Decoder notes that the effect is not clear-cut. It cites a recent study of coding agents that saw almost no measurable gain from more elaborate harnesses, although they consumed up to three times the tokens.
What the evidence cannot say
Each task has one fixed request. Follow-up questions and mid-task corrections from users are not tested. The authors also acknowledge that the AI judges can make mistakes.
The Decoder points to earlier METR research. It reports that project leads' acceptance of AI-generated code ran roughly 24 percentage points below scores on SWE-bench Verified. The authors say older tests are saturated, with scores above 99% when models can run and fix their own code.
Questions to put to your team
First, ask what share of agent-written changes merged without a human edit. Count whole tasks, not checklist items.
Second, test the model and its harness as a pair. Use your own recent change requests, especially ones that were reworded or rolled back.
Third, plan review time for changes that mix layout, design, interaction and content. That is where all ten models lost ground in this test.
Fourth, check that acceptance tests run the application. Reading the code is not proof the feature works.
A score tells you how much was built. Your customers only meet what was finished.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Researchers from Alibaba and several Asian universities (Co-Design Bench authors), via arXiv.
