OpenAI, Google, Microsoft, Anthropic Back One AI Training Opt-Out

Website owners can now ask four of the largest AI developers to keep their pages out of model training with a single two-line entry in robots.txt. OpenAI, Google, Microsoft and Anthropic said on October 30 that they will honor a shared policy token, __usage_ai-training, defined in an Internet-Draft that Anthropic standards lead Kevin Kelley filed with the Internet Engineering Task Force the same day.

Four AI developers agree on one robots.txt token for training

Publishers who wanted their work kept out of training have had to write a separate rule for each company’s crawler, with GPTBot for OpenAI, Google-Extended for Google and ClaudeBot for Anthropic, each documented on a different page. The draft, titled “Machine-Readable Training Preference Signal,” collapses that list into one group that reuses the existing robots.txt grammar without adding any new directives.

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

A disallowed path signals a preference against training and an allowed path signals one in favor. The general wildcard group, “User-Agent: *”, does not count, and a file that never names the token expresses no preference at all. The draft borrows its definition of training from the IETF AI Preferences working group, which describes it as using content to change the learned parameters of a generative AI model.

“This new signal will not replace the controls we already provide through robots.txt.”

OpenAI, announcement, October 30, 2026

The token leaves AI answers and existing models to separate rules

The draft states that the token “is not a directive for crawling behavior.” A provider may still fetch a page that carries the opt-out, and pages pulled live to ground a chatbot answer or an AI search result remain governed by each company’s own crawler tokens, which is why Microsoft is building a second control for grounding. Private licensing agreements can also override the public preference, and the draft itself is an individual submission with no formal standing in the IETF standards process yet.

The timing adds a second layer. On the same day it endorsed the shared signal, OpenAI published a compilation of court briefs arguing that training on copyrighted work is fair use in its dispute with The New York Times. Read together, the two posts show how the largest developers want the question settled: publishers receive a cheaper and more consistent way to say no, while the companies maintain in court that foundation model training requires no permission in the first place.

For publishers, the practical step is to add the shared token now and keep the provider-specific lines in place, because enforcement will arrive company by company through early 2027 and the shared signal cannot pull back anything already learned. The token turns a scattered, crawler-by-crawler chore into one line of text, and its real test begins when Google and Microsoft switch it on and publishers can see whether the four companies interpret that line the same way.