Shared AI Training Opt-Out: What Publishers Need to Know

How does the shared robots.txt rule work?

The draft's whole-site example is two ordinary lines in the site's root robots.txt file. The first line selects the policy token; the second expresses a preference against generative AI training for paths under /:

User-agent: __usage_ai-training
Disallow: /

To express a preference against training only for a section, use a path such as Disallow: /premium/. The draft also illustrates an exception: Disallow: / followed by Allow: /press-kit/ in the same policy group. Its path matching follows the normal longest-match behavior of robots.txt. A site with no __usage_ai-training group expresses no preference through this new mechanism; a wildcard User-agent: * group does not silently become a training preference.

That last distinction is easy to miss. The token looks like a user agent, but the draft calls it a policy token. It does not identify a bot that will arrive in your logs with that HTTP user-agent string. The rule tells participating companies how you prefer content to be used after retrieval. Their actual crawler tokens still decide whether they may fetch a URL.

A preference also is not an access control or a retroactive deletion request. It communicates a choice to providers that adopt and honor it. If a page must stay private, use authentication and appropriate server controls. Do not rely on robots.txt, which is public and cannot secure a page.

Training, crawling, and AI grounding are separate choices

Training means using content to change the learned parameters of a generative model, according to the draft's vocabulary. Crawling means fetching a page. Grounding means drawing on current content to inform an AI-generated answer. These activities can overlap operationally, but the proposed shared signal describes only the training preference. A business that wants its pages found in AI search while declining training must check each provider's search and grounding settings separately.

OpenAI's crawler documentation illustrates the split: GPTBot is associated with training, while OAI-SearchBot is used to surface websites in ChatGPT search. OpenAI says the two settings are independent. Blocking one does not automatically mean blocking the other. Its October announcement says existing crawler controls will continue alongside the common signal.

Google's situation needs particular care. Its Google-Extended documentation says that token governs training future Gemini models and grounding in Gemini Apps and Vertex AI services. Google says Google-Extended does not affect inclusion or ranking in ordinary Google Search. Its new shared-signal announcement says a training opt-out will be interpreted as if Google-Extended had been disallowed when Google-Extended is not explicitly specified. That means the shared signal may have a broader practical effect within those Gemini products than the word “training” alone suggests. Google also documents separate controls for AI features in Google Search.

Microsoft is more explicit about separating the two uses. It plans a shared robots.txt AI Training preference and a distinct, page-level AI Grounding preference for Bing and related Microsoft experiences. Its ai-grounding:disallow setting restricts grounding; ai-grounding:disallowextended allows Microsoft-operated experiences while excluding certain third-party grounding services. Microsoft warns that restricting grounding reduces visibility in AI-generated experiences, even though traditional Bing Search eligibility remains subject to its normal rules.

There is therefore no safe universal statement that “the shared opt-out keeps all AI answers citing me” or “the shared opt-out removes me from AI search.” Product behavior depends on the provider, existing controls, and the rollout. Treat each claim in a settings matrix, not as a single yes-or-no SEO switch.

What each provider has committed to so far

Google's announcement is a plan, not a claim that the token works today. OpenAI has not given a calendar date or complete precedence rules. Microsoft gives the most detailed schedule, but says a published setting becomes active only when it has crawled and discovered it. Those are material limits for a publisher deciding whether it can remove old bot-specific lines.

Microsoft also sets a March 1, 2027 transition deadline for sites relying on NoArchive or NoCache to express training or grounding restrictions. It says the replacement training preference can be placed in robots.txt or, for Bing, on individual pages. Its grounding preference is page-level only, with a temporary Bing Webmaster Tools option for eligible sites awaiting recrawl. Those extra on-page controls are Microsoft-specific; do not copy them into a “universal” checklist for other providers.

The draft author says proprietary policy tokens may be treated as more specific than the shared token, and private agreements may override expressed preferences. Google supplies one concrete precedence rule for Google-Extended. These details are why a staged migration is safer than deleting existing lines the moment a new common line is published.

What does this change for SEO and AI visibility?

The immediate SEO benefit is operational clarity. A team can document its intended AI training policy in one place and, eventually, communicate it to multiple participating providers. The signal is not a ranking tactic, a citation guarantee, or a substitute for discoverable pages. Google says Google-Extended is not a Google Search ranking signal; Microsoft says its usage preferences do not alter normal Bing Search eligibility.

For a content publisher, the core trade-off is not simply “AI on” versus “AI off.” A news site might want timely articles indexed and cited while declining their use to train future foundation models. A software company might want product documentation available for live answers but keep paid course material out of training. A private knowledge base should use access controls before any robots policy. The shared token helps express one of those choices, but it cannot make the underlying content strategy for you.

Measure the outcomes you actually care about. Search Console and Bing Webmaster Tools can show search and AI visibility where their reports support it; analytics can show referral visits; server logs can show fetch activity. None of these directly proves that a model did or did not train on a given URL. A drop in AI citations after changing a rule could come from a separate grounding setting, a crawler block, normal answer variation, or changed demand. Record your policy deployment date so later comparisons have context.

PikaSEO has existing background reading on how AI bots use websites and Cloudflare's crawler controls. Those mechanisms have different scopes. Avoid assuming an edge block, a crawler-specific Disallow, and this new policy token produce identical results.

A practical audit before you edit robots.txt

1. Inventory your live rules. Fetch the actual robots.txt for each relevant host and subdomain. Capture the current version in source control. Use PikaSEO's Robots.txt Checker to inspect structure, then verify behavior against provider documentation. A generic checker may not yet understand the new policy token's use semantics.
2. Write down the policy by purpose. For each content section, decide whether it should be publicly searchable, eligible to inform AI answers, and eligible for model training. Include paid, syndicated, user-generated, and licensed material. Ask content owners before applying a whole-site disallow that may conflict with a partner agreement.
3. Compare current provider controls. Check Google-Extended, GPTBot, OAI-SearchBot, ClaudeBot, and any Microsoft directives you use. Anthropic's crawler guide distinguishes its training, user-directed, and search bots. A single wildcard rule can have very different consequences from a targeted training preference.
4. Test a proposed file in staging. Confirm ordinary Googlebot and Bingbot access to canonical pages, verify that the new group is syntactically separate from existing groups, and check path exceptions against sample URLs. Review technical SEO signals such as canonicals and indexability so a visibility change is not misdiagnosed as a policy effect.
5. Publish and document the decision. Save the exact robots.txt version, affected hosts, date, approver, and intended behavior per provider. For pages needing a Microsoft grounding preference, check the delivered meta tag or HTTP header with an HTTP Header Checker. Revisit the file when the draft or a provider's implementation changes.

Do not add __usage_ai-training merely because it is new. A site happy to allow training does not need an explicit Allow: /; the draft treats an absent group as no expressed preference, and Microsoft says its training default is Allow. If your policy is to opt out, publishing the draft syntax can prepare your file for providers that later support it, but keep the established controls you already rely on until each provider confirms support.

What should publishers watch next?

First, follow the draft's status. It can change before standardization, including how the policy token interacts with more detailed preferences. Second, watch each provider's implementation notice rather than assuming all participants switch on the same day. Google's stated target is 2027, Microsoft's published start is February 1, 2027, and OpenAI has promised a later update.

Third, watch for clarification of precedence and scope. Google has described its Google-Extended exception. OpenAI has said existing crawler controls persist but has not yet published complete shared-signal handling. Microsoft has provided detailed guidance for training and grounding, including its transition from NoArchive and NoCache. Operators with these older directives have a concrete migration project before March 1, 2027.

The safest conclusion on November 4 is modest: a cross-provider training preference is taking shape, and major companies have publicly supported it. Site owners can prepare a deliberate policy and audit their current controls now. They should not claim universal enforcement or an SEO benefit until rollout behavior is observable and documented.

Frequently asked questions

Key takeaways

• The proposed __usage_ai-training group expresses a generative AI training preference through familiar robots.txt syntax.
• It does not itself block crawling, remove a page from search, or establish a universal AI grounding preference.
• Provider adoption is staggered. Keep your current controls and check each company's implementation notice.
• Audit current rules and document the intended policy before editing a production file.

For a broader check, start with the Complete SEO Report and then inspect the live robots.txt and response headers for the pages that matter most.