Quick answer: A new shared robots.txt policy token, __usage_ai-training, is designed to let website owners state one preference about generative-AI training across participating providers. OpenAI says it plans to support the signal in the coming months, Google is targeting 2027, and Microsoft says it will honor the preference as it crawls starting February 1, 2027. It is not fully active across those providers yet, so do not delete existing crawler-specific rules.[1][2][5]
If you want to opt out of AI training now, keep current controls such as GPTBot and Google-Extended while optionally adding the shared token for future support. Keep search and training decisions separate: for example, OpenAI documents that you can allow OAI-SearchBot for ChatGPT search visibility while disallowing GPTBot for model training.[4]
Copy-paste robots.txt options
Option 1: Express a site-wide preference against AI training
User-agent: __usage_ai-training
Disallow: /This states a preference against using all site paths to modify the learned parameters of a generative AI model. The token is a policy preference, not a technical access block, and the underlying specification remains an Internet-Draft that can change.[3]
Option 2: Opt out site-wide but allow a press-kit folder
User-agent: __usage_ai-training
Disallow: /
Allow: /press-kit/The draft supports path-specific preferences and uses the familiar robots.txt longest-match approach. In this example, the site expresses a “no training” preference generally but an “allow training” preference for files under /press-kit/.[3]
Option 3: Keep ChatGPT search visibility while blocking current OpenAI training
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: __usage_ai-training
Disallow: /This is the most practical OpenAI-focused setup for a publisher that wants ChatGPT search eligibility but does not want its content used for foundation-model training. OpenAI treats OAI-SearchBot and GPTBot as independent controls; it also says the new shared signal will not replace its existing controls.[1][4]
What changed on October 30, 2026?
OpenAI and Google announced plans to honor a shared AI-training preference in robots.txt. The proposal uses the policy token __usage_ai-training so a publisher can state a broad choice once rather than maintaining a separate training rule for every participating AI provider.[1][2][3]
Microsoft also announced support, with a stated start date of February 1, 2027 as Bing crawls. Microsoft separates AI training from AI grounding and says these choices do not change eligibility for traditional Bing search results, although blocking grounding can reduce visibility in AI-generated experiences.[5]
The shared token is still work in progress. The IETF document explicitly says Internet-Drafts may be updated, replaced or made obsolete, and this draft expires on May 3, 2027 unless progressed or refreshed. Treat the syntax as an emerging standard, not a permanent legal or technical guarantee.[3]
Training vs search vs grounding
- AI training: using content to modify the learned parameters of a generative AI model. This is what
__usage_ai-trainingaddresses.[3] - Traditional search: crawling, indexing and ranking pages in ordinary search results. The shared training preference does not automatically block traditional search.[3][5]
- AI grounding: using current web content as input to help generate an answer and potentially cite the source. Microsoft provides a separate grounding preference, while OpenAI uses
OAI-SearchBotto manage ChatGPT search inclusion.[4][5] - User-requested visits: OpenAI says
ChatGPT-Usermay visit a page after a user action, and robots.txt rules may not apply to that user-initiated request.[4]
WordPress implementation checklist
- Open your current robots.txt first. Visit
https://yourdomain.com/robots.txt, save a copy, and check whether your host, CDN, security plugin or SEO plugin generates it dynamically. - Choose a policy. Decide separately whether you want training, search visibility and AI grounding. Do not use one vague “block all AI” decision for three different outcomes.
- Keep current provider rules. The shared signal is not fully implemented. Retain any deliberate
GPTBot,OAI-SearchBot,Google-Extendedor other provider-specific entries until official documentation says otherwise.[1][2][4] - Add one shared group. Use only one
User-agent: __usage_ai-traininggroup unless you have tested a path-specific configuration. - Publish through the component that owns robots.txt. This may be a physical file in the site root, an SEO-plugin editor, a hosting panel or a CDN-managed setting. Avoid maintaining competing versions in multiple places.
- Clear relevant caches. Purge the page/CDN cache for robots.txt if your stack caches it.
- Verify the public file. Reload
/robots.txtin a private browser window and with a command-line request. Confirm HTTP 200 and the exact intended text. - Recheck after plugin or CDN changes. A migration, security-plugin update or managed bot setting can overwrite or prepend rules.
Safe verification commands
curl -i https://example.com/robots.txt
curl -s https://example.com/robots.txt | grep -A 5 -i "__usage_ai-training"
curl -s https://example.com/robots.txt | grep -A 3 -E -i "GPTBot|OAI-SearchBot|Google-Extended"Look for an HTTP 200 response, one intended shared-token group, and no accidental Disallow: / under ordinary search crawlers such as Googlebot or Bingbot.
Common mistakes to avoid
- Calling the signal an access block. The draft says the policy token is not a crawling directive. Providers that do not recognize or honor it may ignore it.[3]
- Removing current controls too early. OpenAI and Google described future implementation timelines, not universal immediate enforcement.[1][2]
- Blocking AI search when the goal is only to block training. OpenAI’s
OAI-SearchBotandGPTBotserve different purposes.[4] - Assuming a legal guarantee. The mechanism expresses a machine-readable preference. The draft also notes that private agreements may override preferences.[3]
- Editing the wrong robots.txt. WordPress, an SEO plugin, a CDN and the hosting filesystem can each affect the public response. Verify the live URL, not only an admin-screen preview.
- Publishing conflicting groups. A provider may give its proprietary token precedence over the shared token. Document the intended hierarchy before combining broad and provider-specific rules.[2][3]
Recommended policy for traffic-first publishers
For most publishers that want discoverability but do not want foundation-model training, a cautious policy is:
- Allow normal search crawlers.
- Allow AI search or grounding crawlers only where referral visibility is valuable.
- Disallow current provider-specific training crawlers where that matches your policy.
- Add
__usage_ai-trainingas a future-facing preference. - Review official implementation notes before removing any older rule.
This preserves the possibility of search citations while communicating a separate choice about training. It does not guarantee indexing, citations, traffic, payment or enforcement.
FAQ
Is __usage_ai-training live now?
Not universally. OpenAI says implementation is planned in the coming months, Google is targeting 2027, and Microsoft states February 1, 2027 as it crawls.[1][2][5]
Will the shared signal block AI crawlers?
Not by itself. The draft defines it as a content-use preference, not a crawler-access directive. Use crawler-specific product tokens or server/CDN controls if you also need to limit access.[3]
Does opting out of training remove a site from Google Search?
The shared training preference is separate from ordinary crawling and search indexing. Google says existing company-specific controls continue alongside the shared signal, and Microsoft says its training choice does not change eligibility for traditional Bing search results.[2][5]
Can I allow ChatGPT search but block OpenAI training?
Yes. OpenAI documents OAI-SearchBot for ChatGPT search and GPTBot for training as independent controls.[4]
Should I delete my GPTBot or Google-Extended rules?
No—not yet. OpenAI says its existing controls will continue, and Google says company-specific controls will work alongside the new signal.[1][2]
Can I allow only part of my site for AI training?
Yes. The draft supports path-specific Allow and Disallow rules under the shared policy token.[3]
Last checked: November 1, 2026. This proposal and provider implementation timelines may change.
Sources
- OpenAI: Making it easier for websites to express AI training preferences
- Google Search Central: Supporting a shared AI training control for websites
- IETF Internet-Draft: Machine-Readable Training Preference Signal
- OpenAI Developers: Overview of OpenAI Crawlers
- Microsoft Bing: Your content, your choice: clearer preferences for training and grounding