Confirmed statusThe shared token is a proposal published as an Internet-Draft on October 30, 2026. The announcements describe planned support, not a universal control that every AI provider already honors.
The proposed signal uses an ordinary robots.txt group
The Internet-Draft proposes the policy token `__usage_ai-training` in the `User-Agent` line. Existing `Allow` and `Disallow` rules then express a preference for paths, using the normal longest-match behavior of robots.txt.
A publisher could disallow training for the whole site and allow a press-kit directory. That is more precise than maintaining only one global decision, and participating providers can interpret the same token.
The draft was published on October 30 and expires on May 3, 2027 unless it is updated. An Internet-Draft is working material, not a finished IETF standard. Product teams should track the draft version and each provider’s implementation rather than copy a snippet and declare the policy solved.
| Provider | Published timing | Existing control still relevant |
|---|---|---|
| OpenAI | Implementation planned in coming months | GPTBot and other crawler controls |
| Support planned in 2027 | Google-Extended | |
| Microsoft | Begins honoring on Feb 1, 2027 | Bing crawler rules and new page controls |
| Other providers | No universal date | Provider-specific documentation |
OpenAI and Google are planning support, not switching today
OpenAI says it aims to implement the signal in the coming months and will publish implementation details later. Its October 30 announcement also says existing robots.txt crawler controls will continue to apply and be respected.
Google plans support in 2027. Its Search Central announcement says a shared opt-out will be interpreted as a Google-Extended disallow when Google-Extended is not explicitly specified. A site can still state a separate Google-Extended choice.
For example, a publisher that already blocks GPTBot and Google-Extended should not remove those lines today. It can prepare the shared preference, test syntax and wait until the relevant providers confirm production behavior.
Microsoft separates training from AI grounding
Microsoft says it will begin honoring the shared training preference on February 1, 2027 as Bing recrawls content. Its Webmaster FAQ also adds a page-level training preference and a separate page-level AI Grounding control.
That separation matters. Training concerns use of content to develop generative models. Grounding concerns using current content to inform an answer. A documentation team may make different choices for those uses.
Microsoft says `Allow` for a usage preference does not override a crawler block. A bot still needs technical access to fetch the page. Teams using Microsoft’s legacy `NoArchive` or `NoCache` controls should review the published transition dates instead of assuming those directives cover the new purposes forever.
Training preference, crawl access and visibility are different layers
The shared token communicates a preference about use. A crawler-specific robots rule controls whether a compliant crawler may request paths. A firewall decides whether requests reach the application. Search or assistant selection is a separate outcome again.
A public article might disallow training, allow a search crawler and still never appear in an AI answer. Another page might be allowed for grounding but return 403 because the firewall blocks the user agent. One policy line cannot resolve all four layers.
OpenAI’s crawler documentation distinguishes GPTBot for potential model training, OAI-SearchBot for ChatGPT search and ChatGPT-User for some user-triggered actions. Combine that inventory with a bot firewall policy, then measure actual requests and referrals through AI traffic insights.
Do not treat a voluntary signal as enforcement
The mechanism depends on providers choosing to honor it. It does not stop unidentified scrapers, prove compliance or create a technical access barrier.
A membership site cannot protect paid reports by adding the shared token. Those reports still need authentication and authorization. A public publisher may also need contractual or legal advice for rights questions that robots.txt cannot decide.
The draft deliberately reuses robots.txt syntax, which makes deployment familiar. That simplicity is useful, but it can tempt teams to confuse a declared preference with observed behavior. Keep request logs, provider documentation and policy ownership together.
Audit the current controls before adding the shared token
Start with an inventory rather than a rewrite. Record each relevant crawler, the current path rule, the business reason, the desired training choice, the desired search or grounding choice and the evidence used to verify behavior.
A software company might allow public documentation for search retrieval, disallow training, block account routes at authentication and log assistant requests. The same company may leave its marketing pages open while excluding customer-generated content.
Run a technical SEO audit after any change so canonical pages, search crawlers and sitemap access remain intact. Then revisit the matrix when OpenAI, Google, Microsoft or the IETF draft changes.
- Preserve existing provider-specific rules until support is confirmed.
- Separate training, search, grounding and user-triggered retrieval decisions.
- Keep private content behind real authorization regardless of preference signals.
- Test the published robots.txt file and watch response codes in logs.
- Record the draft version, provider date, policy owner and next review.
Prepare the policy matrix before editing robots.txt
Seuil can help map current crawler rules, firewall behavior, content sensitivity and answer-engine visibility into one reviewable policy. The useful next step is a provider-by-provider matrix with evidence and owners, not a premature replacement of controls that already work.