- Kevin Kelley of Anthropic filed an IETF draft for one robots.txt token that opts a site out of AI training. The draft is not a standard.
- The Verge reports OpenAI, Google, Microsoft and Anthropic say they will adopt it, on different timelines, and the draft's own wording on crawling is weaker than that.
- Check how your robots.txt is written, because a wildcard rule does not match the new token and silence expresses no preference.
From many private signs to one shared sign
Signalling a purpose in robots.txt is not new. The draft notes that some site operators already use proprietary policy tokens, each documented by a single consumer. The Verge adds that sites today must state their AI training preferences to each company separately. What is new is one token meant to be read by every vendor. A patchwork becomes a common signal.
The proposal is an Internet-Draft filed on 30 October by Kevin Kelley of Anthropic with the Internet Engineering Task Force (IETF), the body that publishes internet standards. The Verge reports that four companies have said they will adopt it: OpenAI, Google, Microsoft and Anthropic.
One caution comes first. The datatracker page carries a notice that the IETF has not endorsed the draft and that it has no formal standing in the standards process. It expires on 3 May 2027 unless it is renewed or replaced. Today it is a proposal with big names behind it, not a standard.
How the signal works
A publisher adds a group to robots.txt whose user-agent line names a "policy token": __usage_ai-training. The draft's own example disallows the whole site and allows only a /press-kit/ folder. That tells AI developers the publisher prefers no training, except on the press kit.
The draft adds no new commands and no new grammar. It reuses the existing rules for paths and precedence. Older parsers read the group as belonging to a crawler they do not know, and ignore it. Existing files keep working.
The token is tied to a definition from the IETF's AI preferences working group. Training means using content to adjust what a generative model has learned. Models built only for non-generative work, such as classification, fall outside it.
Two details matter for anyone who runs a site. First, the draft says a wildcard group (User-Agent: *) must not be treated as matching the token. A blanket rule you wrote years ago does not opt you out. Second, if no group names the token, "no preference has been expressed." Silence is not a no.
Who has promised what, and when
The Verge describes the timelines as uneven. Anthropic has already adopted the signal. Google and Microsoft are expected to follow in 2027, the outlet says. OpenAI says it will implement it in the "coming months."
Microsoft will also offer finer controls. Publishers will be able to opt out of AI training or of AI grounding, where a model is pointed to a verified source to reduce hallucinations. The Verge adds that Microsoft says sites will still appear in Bing search results whatever their preferences.
Where the promise is softer than the headline
The Verge says each of the four companies has said its AI crawlers will stay away from a site that shows the token. The draft is more careful. It says the token is "not a directive for crawling behavior." A consumer may use the preference to inform crawling, and may refrain from fetching a page. Nothing in the draft text requires it.
That gap is a question for each vendor: does the token stop collection, or only stop use in training? The answer changes what a publisher is getting.
Two more limits sit in the draft. It says private agreements may override preferences expressed this way, so a licensing deal could change the outcome. And the security section only points to the existing robots.txt rules. It describes no way for a publisher to check that a company honours the signal.
This is why the idea is bigger than the syntax. The proposal separates who may visit from what may be done with the visit, and it does so in a form every vendor can read. That is a sensible split. But a preference line is a request that depends on the reader's promise. It is not a lock.
What leaders should ask their teams
Ask who owns your robots.txt file, and when it was last reviewed. Ask whether your current AI rules were written one vendor at a time.
Decide which content you would allow for training, such as a press kit, and which you would not. The draft supports path-level choices, so the decision can be specific.
Do not assume a wildcard rule covers this. Keep your existing vendor-specific rules in place, since the draft says consumers should read those as they did before.
Ask legal whether any licensing agreement could override the signal. Ask each vendor, in writing, whether the token stops crawling or only training use.
Treat the line as a record of your intent. A clear record of what you asked for is worth having if the promises later prove weaker than they sounded.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: IETF (Internet Engineering Task Force), draft-kelley-aipref-training-preference-signal, authored by Kevin Kelley of Anthropic.
