- METR reviewed Anthropic's August 2026 Risk Report. It agrees direct risk from automated R&D was low on July 15, but says a 2x speedup early-warning line may have been crossed.
- METR says the public report understates the acceleration and leaves out uncertainty from the private version. Redaction was a pre-agreed right, and METR disclosed where it mattered.
- This is a lesson in reading any safety report, not a finding about product safety: separate the rating from the threshold, and ask for ranges and for what was removed.
A safety report can give a verdict and still leave out how close the call was. METR's review of Anthropic's August 2026 Risk Report shows why the margin deserves as much attention as the verdict. The lesson below is this publication's interpretation of one report.
METR is an outside research group. It reviewed the report's section on risks from automated R&D, meaning AI that speeds up or replaces research work inside AI labs. It had access to a private version of the report and other non-public data. It published its review on November 5.
Two different measures
The report uses a "low/medium/high" rating. METR says it understands this rating as the risk that models directly enable the severe harms Anthropic describes. It is not a statement about whether the company crossed a threshold in its Responsible Scaling Policy (RSP). METR says the public report could have made this clearer.
The threshold is a separate, earlier line. Anthropic describes its doubling-of-progress threshold as a potential early warning, not evidence that harm has happened. A company can cross an early-warning line while the direct-harm rating stays low.
A shared verdict, a disputed margin
METR agrees with Anthropic on the rating. It says internally deployed models on July 15 "posed low risk." It adds that significantly greater acceleration would likely be necessary to enable the harm scenarios Anthropic describes.
The disagreement is about the early-warning line. METR says it is plausible the line had been crossed. Its own researchers and Anthropic's relevant staff both lean toward it not having been crossed. METR still believes the public report "understates the level of acceleration that was occurring."
Two conditions, two levels of confidence
Anthropic's policy, in version 3.4, sets two separate tripwires for automated research. Meeting either one counts. The first is models taking over the technical work of the whole research staff. The second is progress running at twice its pre-AI pace because of AI automation.
On the first, METR finds Anthropic's case broadly compelling. Researchers are still needed, even though the company has strong reasons to replace them. METR agrees Anthropic would likely have noticed if models could do the job.
The doubling condition is where METR pushes back. It calls the public conclusion overconfident and says it rests on judgment calls the public text does not explain.
How "less than two" becomes 1.97
Anthropic's public report says progress in its leading indicators has less than doubled, even if all observed speedup were credited to AI. The underlying quantities are redacted in the public text. A reader cannot see what is being measured or recompute the result.
METR could. It combined the private report's current rate with a historical rate from elsewhere in that report and got 1.97. That figure is METR's calculation and is not printed in the private report. METR says readers would take "less than a factor of 2" to mean meaningfully below 2. It suggests "around 2" instead.
METR's own analysis of these indicators gives a range of 1.3 to 2.8, depending on definition and time period. METR says it has substantial uncertainty about how to measure this. Anthropic redacted that discussion.
Redaction here was not a surprise. Anthropic had a right to redact non-public information under the terms of the assessment. METR had a matching right to say whether it did and whether findings depended on it. METR says it described every redaction that mattered to its conclusions. It does show how much a public reader can miss.
Attribution adds a second layer. METR reads the public report as implying fairly high confidence that AI accounts for less than a 2x speedup. METR also notes that the public text does hedge on overall acceleration: it says "we are uncertain and measurement is difficult." The private report calls it "plausible" that the counterfactual acceleration, meaning the speed without AI, exceeded 2x by July 15.
A capability index and a test that stopped working
Anthropic says it is less confident than in earlier reports. Its most concrete task-based tests have "saturated," meaning they no longer register gains in model capability.
METR's concern is what that gap hides. Models may already be capable enough, or nearly so, while problems with alignment, tooling, oversight or elicitation hold back what they actually do. A test every model passes cannot tell these cases apart.
METR also cites AECI. The source describes it only as a capability index with a historical trendline, and this article does not define it further. The public report flags large error bars on the score of a newer model, Model 2. It leaves out the private note that Anthropic expected higher scores from later versions built on the same base. METR says those later Model 2 snapshots did rise on AECI by roughly the margin Claude Mythos 5 gained over Mythos Preview. These appear to be successive Anthropic models, with Mythos Preview first, then Mythos 5, then Model 2.
What this means for readers of safety reports
This review concerns AI speeding up research inside Anthropic. It says nothing about the safety of AI tools that companies buy. Do not read it as a verdict on any product.
The lesson is narrower and still useful. Disclosure quality shapes how much a safety report can tell you. A rating answers one question. A threshold answers another. A range shows how firm either answer is. METR also expects AI's augmentation of R&D to keep growing. It says it cannot rule out substantial further acceleration over the next six months, which could lead to "high" risks. That is METR's view, not a finding that it has happened.
Questions to put to your own team when a vendor or lab hands you a safety report: Does it say what the rating measures, and how that differs from any threshold it cites? Does it give ranges and methods, or only a verdict? Were parts redacted, and did any redaction change a conclusion? Which benchmarks does it lean on, and have they saturated?
A verdict can be right and still leave the margin out. Ask for the margin.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: METR.
