Google plans support for Shared Robots.txt AI Training Control – Unite.AI

Google announced on October 30, 2026, that it plans to support shared control of AI training in robots.txt, giving website owners a single way to express training preferences across multiple AI providers. The commitment came alongside announcements on the same day from Microsoft and OpenAI.

How Google will enforce the control

The plan was laid out in a Google Search Central blog post by John Mueller, a Google Search advocate. The check is based on the work of the AI ​​Preferences working group at the Internet Engineering Task Force (IETF) and is based on Google-Extended, the AI ​​training check that Google says it will offer from 2023.

Instead of managing separate instructions for each AI developer, website owners will be able to use a shared policy token within the robots.txt file to set general preferences across participating AI vendors. When a website uses the token to opt out of training and has not explicitly specified Google-Extended, Google expects to interpret it as if the publisher has prevented Google-Extended. Other AI providers can handle the signal in a similar way, so owners can opt out of all participating providers at once or pick and choose.

According to the post, existing company-specific controls will continue to work alongside Joint Signal, so website owners can express broad choice through Industry Signal, while still tailoring instructions for individual developers through controls like Google-Extended. Websites that specifically allow the Google extension are eligible to appear in replies and receive clicks through rooting in Gemini apps.

Google said the proposal reflects feedback from publishers, civil society groups, regulators and other stakeholders who have called for industry-wide controls on AI training, and that support from major AI vendors would produce a lasting standard that works across the industry. The company also said the proposal is consistent with the EU Code of Conduct for General Purpose AI Model Providers under the EU Artificial Intelligence Law, which recognizes robots.txt as the primary method by which web publishers can express machine-readable preferences. Google aims to support the new control, along with other AI vendors, in 2027 and said it will provide updates on its progress.

OpenAI made a parallel commitment on the same day. OpenAI’s announcement states that the company will honor the common robots.txt signal, aim to implement it in the coming months, and keep existing crawler checks in place and respected. OpenAI said it will join other AI Preferences Working Group participants in the announcement and encouraged other AI vendors to honor the signal.

The IETF draft behind the signal

The mechanism is specified in an Internet draft titled “Machine Readable Training Preference Signal,” written by Kevin Kelley of Anthropic and dated October 30, 2026. The document is intended for the standards track and expires May 3, 2027; according to the IETF datatracker, this is an individual submission that is not approved by the IETF and has no formal position in the standardization process.

According to the draft, a service operator ties usage preferences to paths by writing normal robots.txt groups encoded by a per-purpose policy token, __usage_ai-training, in the User-Agent line. The draft sample group expresses a preference against AI training for an entire site except for paths in /press-kit/. The mechanism introduces no new directive names or new grammar: the path language, longest match precedence, and group structure of the robots.txt standard, RFC 9309, are unchanged, so parsers already deployed on the Internet will ignore policy-token groups they don’t recognize. Wildcard groups, those with User-Agent: *, should not be interpreted as matching the policy token, and a robots.txt file without a __usage_ai-training group expresses no preference.

The token takes its meaning from the definition of AI training in the AIPREF vocabulary: “Using a resource to modify the learned parameters of a generative artificial intelligence model.” The definition excludes training models used exclusively for non-generative tasks, even when the model is capable of generative tasks. The draft also emphasizes that the token indicates a preference on content use rather than a directive on scanning behavior, although consumers can use preferences to inform scanning; specific expressions should take precedence over general ones, and private agreements may prevail over expressed preferences.

Kelley announced the draft IETF ai-control mailing list on October 30, 2026, writing that Anthropic has already begun implementing support for it. He described the work as a complement to Anthropic’s existing crawler controls rather than a replacement.

Microsoft adoption and transition dates

Microsoft announced its adoption the same day in a Bing blog post by Krishna Madhavan, Principal Product Manager at Microsoft Bing. Microsoft said it was the first company to provide controls to publishers for AI training and grounding, pointing to the NoArchive and NoCache tags introduced in 2023.

Microsoft is taking the shared AI training preference for sites and locations and extending it to the page level via meta tags and HTTP response headers, while adding a separate page-level AI grounding preference. In addition to Allow and Disallow, the grounding preference adds a new DisallowExtended option that limits the use of grounding to Microsoft-managed experiences such as AI Mode, AI Overview, Copilot, and Microsoft 365 while excluding grounding services provided by Microsoft to third parties, such as Web IQ and grounding via API access in Foundry. Both preferences are automatically set to Allow when no preferences are published. Microsoft warned that selecting Disallow or DisallowExtended for grounding reduces brand visibility in AI experiences; preferences do not change eligibility for traditional Bing search results and do not establish licensing or payment agreements.

The new preferences will replace NoArchive and NoCache for training and grounding and will go into effect on February 1, 2027, as scanned by Microsoft. Temporary grounding transition preferences will open in Bing Webmaster Tools on December 15, 2026 for sites using legacy tags on or after that date. Such sites must publish their new preferences and register a temporary grounding preference before March 1, 2027, after which NoArchive and NoCache will no longer be read as training or grounding preferences. Temporary preferences remain in effect until Microsoft recrawls a page and discovers the new base preference or until September 30, 2027, whichever comes first.