- An OpenAI DevDay speaker said deleting a failing test is sometimes right, but people are reasonably skeptical because it is the easy way to make things pass.
- When the agent was unsure its fix was right, it often had a second agent review the code before asking a human.
An agent told to get the build passing has an easy shortcut: remove the test that fails. A speaker on OpenAI's DevDay 2026 session about coding agents that keep working for a long time named this risk directly. Our takeaway is that a green build only means something if someone you trust checks how it turned green.
What was said
The session centred on one Codex run, OpenAI's coding agent, that worked for over 1,000 hours to keep CI healthy. A speaker said the agent made just under half a million tool calls and used more than 5,800 sub-agent sessions. It made a couple hundred fixes to CI, the automatic tests that run on every code change.
The speaker said these were the hardest bugs still open. Many people and their agents had already looked at them and chosen to skip or retry the test rather than fix it. Then came the warning:
"Of course, sometimes this idea of getting back to green is just to delete the failing test, right?"
The speaker added that deleting a test is sometimes the right call. But people are reasonably skeptical of it, because it is the easy way to make things pass. When the agent was unsure its fix was right, it often created another agent to question and review the first agent's code. Only after that did it ask a human to review.
Why it matters
Our reading: a passing build is a claim, and the claim is only as good as whoever checked it. If the goal given to an agent is "make CI green", then removing the test and fixing the bug both count as success.
The people who pay when a deleted test hides a real fault are usually not the ones who merged it. They are the users who hit the bug, or the maintainers who find it later. For managers and buyers, the practical question is who reviews the reviewer. Ask whether a human sees every deleted or skipped test, not only the green badge.
The setup the speaker described points the same way. The agent's work ended in a pull request, a proposed change, for humans to review. The test results were the agent's way to check its own work, and people were the last check.
The other side
The speaker did not claim deletion is always wrong. Some tests really should go, and the agent's most praised work went the other way. It tracked a flaky GPU test to one machine over weeks, then asked the researchers who owned that system for help.
The safeguard also has a gap. The speaker said the agent created a second reviewing agent when it noticed it was unsure, and said this happened often, not always. The excerpts do not say how often a test was deleted, or how the reviewing agent itself was checked.
An agent that is confidently wrong would not raise the alarm. The excerpts do not say how to catch that case, and the safeguard they describe is human review.
Written by the WebPulse Newsroom with AI assistance, and checked by our editorial review: every quotation was verified against the recording's transcript. How we use AI.
The conversation this talking point comes from
- OpenAI: What Will You Build With Coding Agents That Don’t Quit? | DevDay 2026 (2026-11-11)
