- Researchers on AI Engineer found that coding agents which complete tasks often cannot reverse their own side effects when something goes wrong, because Git covers only the repository.
- In their tests, 47% of normally successful runs recovered under injected failure, and 197 of 478 capability gains left recovery defects.
A coding agent that completes a task is not necessarily one that can repair things when the task goes wrong. Researchers Tanmay Sah and Dolly Sah made this case on the AI Engineer show. Their evidence suggests buyers of coding agents should ask how an agent recovers before they ask how much it can do.
What was said
The Sahs ran a benchmark called UndoBench. It used 2,880 paired trials, each comparing a normal run with a run where a failure was injected. Under normal conditions the agent succeeded 84% of the time. Of the runs that had succeeded normally, 47% recovered once a failure was injected. The researchers noted that the two figures have different denominators, so they are not a simple before-and-after.
They gave a concrete case. An agent asks an outside system to create a ticket. The server creates ticket 781, but the confirmation gets lost. The agent sees a timeout and sends the request again. Now there are two tickets. As a speaker on the show put it, "A timeout tells us that the response didn't arrive, not that the operation failed." Retrying blindly led to doubled actions in 54% of comparable UndoBench trials.
A second project, EvoUndo, looked at agents that change themselves. Of 600 tasks, 478 made the agent more capable. In 197 of those, about 42%, the change also left recovery defects. A speaker on the show summed it up: "an agent can become more capable while becoming less recoverable."
Why it matters
Our reading: a demo shows an agent finishing a job. It rarely shows what happens when the job half-fails. Git records changes to the repository, but the Sahs pointed to effects it does not cover, such as external API calls, runtime configuration and notifications.
The researchers argued that recovery should be part of the decision to admit an agent into production, not something planned after deployment. For buyers of coding agents, that suggests questions. What does the agent do after a timeout? Does it record the original state first? Who checks that the restoration worked?
The other side
These are the researchers' own benchmarks, built around injected failures. The talk does not show how the numbers carry over to products in real use. It also does not say which agents or products were tested, so the findings should not be read as a verdict on the market.
The Sahs also reported that recovery can be improved. Prompting agents to repair the 197 failures did not work. But when the agent was given precise information about what had changed, recovery rose from zero to 38 of 48 cases. With richer undo operations, it rose from zero to 142 of 143. Those came from different subgroups, and the fix is the researchers' own project.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- AI Engineer: Your Coding Agent Can't Undo Its Own Mistakes - Tanmay Sah & Dolly Sah (2026-11-08)
