- Vals AI found that AI agent teams cost 1.8 to 5.1 times as much as single agents on a web-app benchmark, and only one of four score gains was statistically significant.
- The clear win came where teams built and cross-checked work that single agents skipped, though the study cannot separate extra building from independent review.
- Before buying team setups, ask what they add, and compare against one agent at higher reasoning effort.
Managers know that adding people to a project adds meetings, hand-offs and mistakes at the seams. AI vendors now sell the same bet: let one agent hire several others. New research from the evaluation company Vals AI tests that bet. The answer is mixed, and the mixed part is the useful part.
What Vals AI measured
Vals AI used its Vibe Code Bench, which asks an AI to build a complete web application from a product spec. Finished apps are launched in containers. Automated browser agents then work through each app's user flows and count what passes.
The study covered 50 apps and two models, GPT 6 Sol and Claude Opus 5.5. Each model was tried solo and as a team, at medium and at max reasoning effort. That makes eight setups, each run once.
In a team, a lead agent could hand work to as many as five subagents at once. A team's bill ran from 1.8 to 5.1 times a solo agent's. Only one of the four team-versus-solo comparisons showed a statistically significant score gain.
Where the extra money goes
Each subagent keeps its own working memory, which holds the spec and its progress. That memory is re-sent to the model on every call. The researchers say most of the extra spend is this cached input.
The volume is large. In Opus's max-effort team, the helper agents re-read a median of 224 million cached tokens per app. The lead agent read 17 million. A solo agent read 55 million. Each added worker adds more repeated reading.
What the team actually bought
The one clear win came from Sol at medium effort. Its team beat its solo agent on 24 of the 50 apps, trailed on 11 and tied on 15. The average edge was about 15 points on harder apps and about 1 point on easier ones.
Vals AI looked closely at nine apps where the team gained 20 points or more. In those cases the solo agent reported that testing had passed. Yet parts of the spec were missing, and some workflows had not been exercised. The team built the missing features. Its lead also compared the separately built parts against each other. It caught an interface sending fields that did not match the database, and a permission rule that blocked a step.
One reading is that the gain came from two things together: more of the spec got built, and more of it got cross-checked. The study cannot split the two. It had one run per setup, and the team prompt bundled a delegation instruction with the subagents. For Sol, the logs show the lead's commands but not what the subagents did. The lead did much of the testing itself.
The pattern suggests a multi-agent setup may work less like a bigger workforce and more like added coverage plus a second pair of eyes. If a single agent already covers and checks its work well, the team has less to add.
Paying for effort instead
Higher reasoning effort was an alternative, and it did not always lose. For Sol, max effort raised the single agent's score by 11.4 points, more than the team's 7.3. The max-effort single agent scored 89.0% against 84.9% for the medium-effort team. That gap was not statistically significant. It cost 1.6 times as much and took three times as long.
Opus told a different story. At medium effort its solo agent scored 91.5% and cost $4.08. At max effort the solo agent scored 89.8% and cost $23.77.
Opus's max-effort team posted the study's highest score, 93.2%. It cost about 30 times as much as the medium-effort solo agent. The 1.6-point gap between them was not statistically significant, so it could be chance.
Speed gains were uneven
Sol's max-effort team finished in 28 minutes against 38 for its solo agent. At medium effort the gap was small: 11.9 minutes against 12.4, at 2.5 times the cost. Opus's teams were slower. At max effort they ran up to 2.3 times as long as solo runs.
Opus's lead wrote a shared contract file first, then ran builders and testers in waves, each wave waiting for its slowest member. Longer runs did not score higher. Runtime had no correlation with the Opus max team's gain.
The Decoder, which covered the study, reports that OpenAI researcher Noam Brown said on the Dwarkesh Podcast that multi-agent systems mainly buy speed, not quality. In his example, four agents worked twice as fast at twice the cost. That view matches only part of the Vals data. Sol's max-effort team was faster. Sol's medium-effort team was barely faster yet still gained on score. Opus's teams were slower. Treat his remark as context, not as confirmation.
Limits of the evidence
This is one benchmark of full-stack web apps. Each of the eight setups ran once. The two models ran through different tools, so comparing them mixes the model with its harness. Team runs also received a delegation instruction, so each test compares subagents plus a prompt against neither. The researchers say the results may not carry over to other work.
Questions to put to your team
First, ask what the team is meant to buy: speed, wider coverage or independent checking. Second, ask for a single agent at higher effort as the baseline, with its cost and score alongside. Third, ask whether agent-reported test results are verified by something else. In the Sol runs, "testing passed" did not mean the work was complete. Fourth, ask for cost per finished task, not per call, because repeated context is where team bills grow.
Agents multiply output, and they can also multiply the places where work is quietly skipped. Pay for coverage and checking, and measure what they return.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Vals AI.
