Google, Microsoft and OpenAI have announced plans to honour a single shared signal in robots.txt that lets a website state its AI training preference once, for every participating provider, instead of writing separate rules for each company’s crawler.
The signal is a policy token, __usage_ai-training, from an Internet Draft submitted on 30 October to the Internet Engineering Task Force’s AI Preferences working group by Kevin Kelley of Anthropic. A website would add an ordinary robots.txt group under that token, using Allow and Disallow lines as usual — for example, Disallow: / to opt out a whole site, with exceptions for particular folders.
The idea is deliberately narrow. The draft adds no new robots.txt directive, it does not control crawling itself, and a file with no __usage_ai-training group signals no preference at all. Nor does it settle licensing or payment: Microsoft has said the new preferences do not establish licensing or payment arrangements. It covers training — using content to change a generative model’s learned parameters — rather than search indexing or AI answers that quote a page.
Crucially, none of the three companies supports the line yet, and their timelines and handling differ. Google aims to support it in 2027, with its crawler documentation pointing to early 2027; where a site opts out through the shared line without specifying Google-Extended, Google plans to treat it as if the publisher had disallowed Google-Extended, a control covering future Gemini model training and grounding in Gemini Apps and on Vertex AI.
Microsoft is the firmest on timing. It will begin honouring the new preferences as it crawls from 1 February 2027, and it is adding a separate page-level AI Grounding preference, which can also be set by meta tag or HTTP header. Microsoft will stop recognising its NoArchive and NoCache tags as training or grounding signals after 1 March 2027, so publishers still using those tags and wanting to stay out of grounding must record a temporary preference in Bing Webmaster Tools, a setting that opens on 15 December and is available only to sites still using the tags on that date.
OpenAI says it aims to honour the signal in the coming months, while its current robots.txt policy stays in effect. What happens when a shared-line rule conflicts with a GPTBot rule is not yet spelled out in OpenAI’s post or crawler documentation, as checked on 30 October. Anthropic, whose employee wrote the draft, has begun development on support, but had published no newsroom announcement about the signal as of 30 October — so its backing should be treated as in progress, not yet confirmed as a launch commitment.
The draft itself is also provisional. It is listed in the IETF’s document tracker as an individual draft, which does not represent official IETF standardisation activity, and it expires on 3 May 2027 unless a new version replaces it. Kelley plans to propose his design as a change to the working group’s Content-Usage draft.
For publishers, the promise is simplification: one group instead of a stack of per-company training rules. The catch is that the same opt-out will not mean exactly the same thing everywhere. Microsoft keeps training and grounding separate, while both sit under Google-Extended at Google, and Google’s documentation describes no way to keep content out of Gemini training while still allowing Gemini Apps to use it for grounding. Publishers should therefore check each provider’s current documentation before relying on the new rule — and should not assume it is already being enforced anywhere.
Note on sourcing: the company timelines and handling described here rest on Search Engine Journal’s 1 November reporting of the companies’ posts; Anthropic’s position is unconfirmed as a formal announcement, as noted above.
Sources: Search Engine Journal, “Google, Microsoft & OpenAI Plan To Honor Shared AI Training Opt-Out”, 1 November 2026, https://www.searchenginejournal.com/google-microsoft-openai-shared-ai-training-opt-out/595387/
