OpenAI will honor the Shared Robots.txt signal for AI training preferences – Unite.AI

OpenAI announced on October 30, 2026 that it will honor a shared signal in robots.txt, the file that websites use to express preferences for how their content is used, so that site operators can communicate a single AI training preference to multiple participating vendors at once.

A voluntary signal shared between suppliers

In its announcement, OpenAI described the initiative as voluntary and said it solves an existing burden: Until now, website operators had to identify individual crawlers and maintain separate rules for each provider. A shared signal, the company said, allows publishers and website operators to communicate the same preference to multiple participating providers at the same time. OpenAI said it plans to deploy the signal in the coming months and will share another update with implementation details when it does.

The company said the new signal will not replace the controls already provided via robots.txt, which will continue to apply and be respected. OpenAI interpreted the move as drawing on the ongoing work of the Internet Engineering Task Force’s AI Preferences Working Group, said it would join other working group participants in making the announcement, and encouraged other AI vendors to honor the signal. The company described the step as a move toward an open web where people can publish and discover information and website owners have clearer and easier ways to communicate their choices about how their content is used, and said it will continue to adapt its practices as conversations about standards evolve.

How the signal works

The signal is defined in an Individual Internet-Draft titled Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00), written by Kevin Kelley of Anthropic, dated October 30, 2026, and expiring May 3, 2027. The IETF datatracker notes that an Individual Internet-Draft is not approved by the IETF and has no formal standing in the IETF Standards Process.

According to the draft, a service operator ties usage preferences, such as “no AI training,” to paths by writing normal robots.txt groups encoded by a per-purpose policy token in the User-Agent line. The draft introduces no new directive names or new grammar: the path language, longest match precedence, and group structure of the robots.txt standard published as RFC 9309 remain unchanged. As a result, the draft states, existing parsers deployed across the Internet will continue to interpret traffic-handling signals as they did previously and will ignore policy-token groups as they would any group that names a user agent that it does not recognize.

The draft defines the __usage_ai-training cross-vendor policy token. His example group pairs the line “User-Agent: __usage_ai-training” with “Disallow: /” and “Allow: /press-kit/”, which expresses a preference against AI training for all paths except those in /press-kit/. Wildcard groups, meaning “User-Agent: *”, should not be interpreted as matching the policy token, and if no groups in a file contain the token, no preference has been expressed.

The draft also states that the policy token is not a directive for scanning behavior. Content usage preferences and crawler traffic management signals are independent and orthogonal, it is claimed, and private agreements can override preferences expressed through the mechanism. Its stated goals include path-specific constraints, minimum distance from RFC 9309, no large-scale rule duplication, unchanged behavior for parsers that do not implement the mechanism, and consistent interpretation across consumers.

The vocabulary behind the signal

The meaning of the token tracks the working group’s draft vocabulary, A Vocabulary For Expressing AI Usage Preferences (draft-ietf-aipref-vocab-08), written by Paul Keller of Open Future and Martin Thomson of Mozilla and last updated in September 2026 as a working group document intended as a proposed standard. This vocabulary defines AI training as using a resource to modify the learned parameters of a generative AI model, excluding training of models used exclusively for non-generative tasks, and defines a generative AI model as one used, or made available for use, to generate synthetic content in one or more modalities.

The vocabulary also defines a separate AI usage category, which covers assets used as input to a generative AI model rather than provided directly by a user, and a search category for applications whose primary purpose is to select assets and direct users to their locations. Each category is assigned an allowed, disallowed, or unknown preference value, and where instructions conflict, the most restrictive preference applies. The draft vocabulary states that it does not guarantee that preferences will be followed and leaves the choice up to the recipient whether to follow them, and a note to readers warns that its contents do not yet reflect the consensus of the working group. One section, whose inclusion is still charted as an open question, states that respecting training preferences includes carrying preferences associated with training resources along with a template when it is deployed.

The training signal draft credits name working group chairs Mark Nottingham of Cloudflare and Suresh Krishnan of Cisco, vocabulary editors Thomson and Keller, and area managing director Mike Bishop of Akamai. It also lists individual participants, with affiliations noted for identification only because IETF participants contribute as individuals, from organizations including OpenAI, Google, Microsoft, Meta, Cloudflare, Common Crawl Foundation, News Corp, Financial Times, BBC, Electronic Frontier Foundation, and Creative Commons.

OpenAI’s existing crawler controls

OpenAI’s crawler documentation, which the company says will remain in place alongside the new signal, describes GPTBot as the agent that crawls content that can be used in training its underlying generative AI models; disallow GPTBot indicates that the content of a site should not be used for such training. OAI-SearchBot displays websites in ChatGPT’s search functionality, and sites that disable it will not appear in ChatGPT’s search responses, although they may still appear as navigation links. OAI-AdsBot validates the safety of web pages sent as ads on ChatGPT, and the data collected is not used to train basic generative AI models. ChatGPT-User handles page visits triggered by user actions in ChatGPT rather than automatic scanning, and the documentation notes that robots.txt file rules may not apply to such visits.

Each setting is independent of the others, according to the documentation, so a webmaster can allow OAI-SearchBot to appear in search results while preventing GPTBot from training. Search changes can take approximately 24 hours after a site’s robots.txt file is updated. OpenAI said it will share implementation details of the shared signal when it rolls it out in the coming months.