Cohere's tests: teams of coding agents duplicate work, overreach and cost far more

Teams of AI coding agents can get more done than one agent. In Cohere's tests, though, they also repeated each other's work, strayed beyond their jobs and cost far more. Nouha Dziri, a senior research scientist at Cohere, reported the results on the AI Engineer show. She also called security for autonomous agent swarms a question the field has yet to answer.

What was said

Dziri's team gave a very hard programming task to a group of agents. They expected tidy groups, shared work and good coordination. That is not what happened. Dziri said each agent chose to go solo, and in her words, "everyone went to build their own version." When the agents did talk, she said, much of it was off topic, and some of it got in the way. She said 84% of messages went straight to the lead, so nobody could see who was doing what.

Extra background made workers overreach. They grew keen to add features nobody had asked for. She also described an asynchronous lead, one that does not wait for its workers. It guessed ahead and sometimes ignored their reports. Sometimes it decided the task was finished while workers were still running, and their work was lost. A lead that blocks and waits did not have this problem.

Cost was the other problem. She said a team costs six to seven times more than a single agent. In one run, about 830 further delegation requests cost around $500 and did not improve performance. Later she noted that "we now have teams of agents who can work autonomously 24/7 without supervision." She asked how to secure systems and infrastructure.

Why it matters

Our reading: the failures she describes are ordinary management failures, now running at machine speed. Duplicated effort, scope creep and early sign-off are familiar in human teams. Agents can repeat them around the clock, and each wasted step is billed.

For anyone buying or running agent tools, the lesson is to check where the money goes. The early-finish problem faded once the lead was told how to coordinate. That fix costs something, either in prompt-writing time or in slower, pricier runs. A budget cap and a human check on what "done" means look sensible, given her findings.

The other side

These were research experiments with a minimal-prompt, top-down team design. They used two coding benchmarks, ProgramBench and Commit0. She said the best team design depends on the task. Extra workers helped at the planning stage in one and at implementation in the other.

Dziri also pointed to savings. The show notes say a strong lead with cheaper planning workers gets near an all-frontier team for about 70% less. On security she offered no solution.