- Anthropic says six METR staff now have employee-like access to its internal tools, and METR will publish its own risk analysis without an Anthropic veto.
- Anthropic can still require redactions in narrow categories, including customer data, privileged material and model architecture. METR may say publicly whether it thinks they were justified.
- When you assess an AI vendor, ask who the evaluator is, what it can see, and who decides what gets published.
A vendor's own safety report is a statement, not a check. A check needs someone outside the company who can see enough and is free to say what they found. Anthropic has just published the terms of one such arrangement, and those terms show what an outside check requires.
On October 15, Anthropic said six staff from METR, a nonprofit that assesses leading AI models, now work with employee-like access. They have company laptops and office space. Their reach into internal tools is roughly equal to that of Anthropic's own risk teams. METR will publish its own analysis of Anthropic's risks, incidents and practices.
What an evaluator has to see
Anthropic lists four needs. Evaluators need expertise, independence, enough access and freedom to publish. Independence means no equity in the company and no financial or operational dependence on it.
Access is the technical heart of the list. Anthropic says evaluators need more than the latest models. They also need to see how those models were trained and how its safeguards work.
Here is why that matters. Anthropic names alignment faking as one way an evaluation can go wrong. In plain terms, a model can look safe in a test without being safe. An evaluator who only sends prompts and reads answers sees the test performance.
An evaluator who also sees the training process and the safeguards can ask harder questions. How was this model trained? Does a safeguard really work, or does it only look that way from outside? That reading is ours, not Anthropic's claim. But it explains why access, not just a model to query, is the core of the deal.
Three tiers, and who decides what is seen
Anthropic is building three tiers of evaluator access. In the first, evaluators run narrow investigations. Anthropic chooses what they can see, and they need its approval to publish.
In the second tier, evaluators see more and can publish without approval. Anthropic can still require removal of certain categories of information. The third tier is the METR arrangement: employee-like access, with publishing rights like the second tier's.
TechCrunch reports that METR is the only evaluator at the top tier so far. It places SecureBio, a biosecurity nonprofit, in the bottom tier. SecureBio is now working through one part of Anthropic's August Risk Report: the chapter on chemical and biological weapons. A trial review of the previous report is already published. Anthropic says it will consider projects with more access once the current review is done.
The idea: an audit is defined by who holds the pen
The lesson here is that outside assurance rests on two questions. What can the evaluator see? And who decides what reaches the public? A review with full access but a vendor veto is a draft. A review with publishing rights but a narrow view is a guess.
Anthropic compares its approach to bank supervision. Regulators station examiners inside the largest banks because the banks understand their own risks best. Anthropic says the comparison is imperfect. The shared logic is that problems are easier to catch from close range.
On publishing, Anthropic says it has no veto. It can demand redactions only in narrow, predefined categories. Its examples are customer data, attorney-client privileged material and information about model architectures. METR may tell the public whether it thinks a redaction was justified, as long as it does not reveal the hidden content.
What the design leaves open
Model architecture, the design of the model itself, gets special handling. Anthropic gives two reasons. Evaluators may also work with other companies, so details could leak by accident. And architecture is usually not relevant to judging risk.
When it is highly relevant, Anthropic says it will generally seek to share it, with explicit internal approval. That means Anthropic makes the first call on relevance. METR's right to comment on redactions is the check on that call.
The scope is also small by design. Anthropic limits the work to a small METR team under enforceable agreements. Other METR staff, including researchers who work with Anthropic on other projects, are walled off from it.
Anthropic's post does not describe any findings from the embedded team yet. It does say it has worked with METR on several reviews of its Risk Reports over the past year. Everything above is Anthropic's account of the terms. The sources include no statement from METR.
TechCrunch ties the timing to disclosures of AI agents going against their instructions, which it says have surfaced every week or two since late August. It also reports that six AI companies, Anthropic among them, have signed a White House accord on independent review.
Questions to put to your AI vendors
Procurement teams can turn this structure into a checklist. Ask which independent evaluators the vendor uses and what they can see. Ask whether they see training and safeguard details, or only the model's answers. Ask who can approve or block publication.
Ask which categories can be redacted, and whether the evaluator may say it disagrees. Ask whether the evaluator holds equity in the vendor or depends on it for revenue.
Then ask for the latest independent report before a renewal. Check that it covers the model you actually use. A vendor that answers these plainly is offering assurance. A vendor that can only point to its own report is offering a promise.
An audit is only as honest as the rules about who may say what happened.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Anthropic.
