- METR and Redwood Research tested OpenAI's internal system for monitoring its AI agents and found gaps in coverage and several ways an agent could avoid review.
- At the time, the monitor ran when an agent handed back control, so while compaction went unmonitored an agent could in principle delay review by continuing to act.
- Leaders buying agent oversight should ask what share of activity is monitored, how that share is measured, and whether the agent can influence when the monitor runs.
Saying you monitor your AI agents is easy. Showing how much of their work the monitor sees is hard. A three-week outside test of OpenAI's internal system shows why the second question matters more.
What the testers did
Between late July and mid-August, one person from METR and one from Redwood Research were given deep access to the tooling OpenAI uses to catch its own AI agents behaving badly. Their job was to stress-test it. They could read OpenAI's internal code, the monitor's prompts and settings, and internal documents. They also interviewed about a dozen researchers and engineers. They worked on OpenAI-issued devices that stayed on the premises.
The work produced a 61-page internal report. METR published a short public post on October 22.
The public post has limits. It describes a pilot, not a full transparency exercise. OpenAI could redact anything it judged sensitive. METR did not have the right to describe those redactions. METR says it endorses each finding it presents.
The findings cover the system as it stood on and before July 21. OpenAI was expanding the system during the test. The test did not establish today's coverage.
How much activity was watched
METR estimated that 75–94% of internal agent traffic through OpenAI's Responses API was eligible for monitoring. That service powers Codex, among other things. The figure applies to traffic from GPT-5.5 and stronger models. METR says the true value could be higher or lower. For the part used by internal Codex agents, METR put the figure at 99.2%.
Two limits apply. The estimate leaves out agents on OpenAI's internal research inference service, because METR could not size that traffic. Also, "eligible" means intended for monitoring. Errors, load shedding or bugs could still keep some of it from reaching the monitor. OpenAI showed METR figures suggesting the monitor failed on under 0.2% of requests.
Other gaps came up. Reinforcement-learning training runs were not covered when the test began. OpenAI told the testers a subset had been monitored from March to June. Compaction events, when an agent condenses its long working history, were also outside the monitor's reach.
METR's rough estimate is that around a tenth of eligible internal Codex traffic skipped review from May to July 2026. The exact share is uncertain. The logs had not recorded how many monitor calls compaction triggered. OpenAI staff spotted the problem before the test and fixed it during the test.
The monitor ran on the agent's schedule
The most useful detail is a design choice. At the time, the monitor fired when an agent handed control back to the user. While compaction events went unmonitored, an agent could keep review waiting by simply continuing to act. METR could not check whether any agent had done this.
Think of an auditor who checks the books only when the bookkeeper hands them over. Coverage then depends on the very behavior you want to catch. The lesson here is that a monitor needs its own clock, one the watched system cannot influence.
Other ways around the monitor
METR listed more findings, each with caveats. One path would let a Codex agent step outside its sandbox and past Auto-Review. In principle, the agent could then do anything the host user could do.
Another exploit could keep hundreds of recent actions out of monitoring, including the exploit's own actions. OpenAI redacted the details.
Agents could likely craft inputs that make the monitor fail. The record of an agent's steps is written out differently for the agent and for the monitor, and that gap can be abused. A failed monitor does not send the case to a human reviewer.
Two settings that the client controls could switch off monitoring for new agents. Using them could itself be caught. Two internal tools thought to be read-only had a simple exploit. METR did not test whether it worked in practice.
There was good news too. METR checked what the monitor is shown. Apart from a few small exceptions, it gets the full record of the agent's steps. Its prompts also look for scheming, sabotage, data theft, deception and prompt injection, among other risks.
What OpenAI says now
OpenAI has told METR that all the vulnerabilities affecting its production safeguards are patched. It also described significant changes to the monitor's intended coverage. METR says it has not evaluated either claim.
Questions to put to your team
This source concerns OpenAI's internal systems, not yours. The questions still apply to any organization that lets AI agents act on its behalf.
First, what share of agent activity does the monitor see, and who measures it? METR advises keeping a full account of all inference and checking it against the monitor's own logs. Second, can an agent influence whether or when the monitor runs? METR suggests checking every N messages or minutes, not at the end of a turn.
Third, who owns incident response, and are the logs tamper-proof? Fourth, when did someone outside the team last try to break the monitor?
A control you cannot count is a hope. Ask first for the share of activity your monitor does not see.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: METR.
