AI crawler training opt-out in one robots.txt rule

What OpenAI, Google and Microsoft announced

OpenAI said on October 30, 2026 that it will honor a common signal in robots.txt that tells AI providers whether a site's content may be used to train generative models. Google and Microsoft published matching plans the same day. A single robots.txt group will soon carry a site's refusal of AI training to several providers at once, yet none of the three companies reads it so far.

Until now, an AI training opt-out meant looking up each company's crawler and writing a separate rule for it. OpenAI describes the shared signal as a voluntary effort that lets one preference reach every participating provider.

When each company will honor the signal

Microsoft is the earliest company with a date, February 1, 2027, while Google targets early 2027 and OpenAI only says "in the coming months." Here is how the three announcements compare.

Microsoft will apply the preference to pages it crawls from that date. Google's blog post says "in 2027," and its crawler documentation narrows that to early 2027. OpenAI promises implementation details when it ships.

Anthropic says it has already started

The draft's author, Kevin Kelley, works at Anthropic. Kevin Kelley wrote on the IETF mailing list that Anthropic has begun building support, though rolling it out across its services will take time. He added that the signal complements Anthropic's existing crawler controls rather than replacing them, and that its interpretation may change as the standard develops.

The rule is an IETF draft, not a standard

The IETF is the body that develops internet standards, and its AI Preferences working group is where the signal is being discussed. The token is defined in an individual Internet-Draft that expires on May 3, 2027, so its details can still change and only companies that opt in will follow it. Internet-Drafts are valid for six months at most and can be revised or replaced at any time.

How to write the shared AI training opt-out

The three lines in a shared training group

User-Agent

Write __usage_ai-training. It names a use, AI training, rather than a crawler

Disallow

Paths listed here carry a preference against AI training

Allow

Paths listed here carry a preference in favor of AI training

The draft calls the new value a policy token: a User-Agent value that names a purpose instead of a crawler. You add a group whose User-Agent line is __usage_ai-training instead of a crawler name, then use Disallow to refuse training and Allow to permit it.

Paths, wildcards inside paths and line order follow the same RFC 9309 rules you already use, and no new directives are added. A parser that has never heard of the token treats the group like one for an unknown bot and skips it.

Block AI training across your whole site

A two-line group with "Disallow: /" under the token expresses a preference against training for every page on the site. Microsoft's announcement uses exactly this example.

User-Agent: __usage_ai-training
Disallow: /

To cover only part of a site, list a path such as Disallow: /example-path/ instead. Microsoft notes that Allow is the default, so a site that wants its pages used for training does not need to add anything.

Allow training for a single folder

Adding a line such as "Allow: /press-kit/" carves one folder out of a site-wide Disallow because the longer path match wins. The draft uses a press kit as its example, a folder a company would want AI systems to learn from.

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Every URL under /press-kit/ matches both lines, and the Allow line is the longer match, so those pages carry a preference in favor of training. Everything else on the site stays opted out.

Why a wildcard group does not count

Rules in a wildcard group are never read as a training preference, so a site without its own __usage_ai-training group has expressed no preference at all. That holds even if the wildcard group already says "Disallow: /".

Kelley described the exclusion as deliberate: no robots.txt file published before the draft should be read as taking a position on AI training. If you already block everything under a wildcard and also want to opt out of training, you still need to add the named group.

What the opt-out does not change

What the shared group affects, and what it leaves alone

AI training

Your preference reaches each provider once it supports the token

Crawling

Not limited. Use a group for each crawler's own name to block fetches

Company rules

GPTBot, Google-Extended and ClaudeBot rules keep working

Earlier fetches

Judged by the robots.txt in place when each page was fetched

Licensing deals

A private agreement can override the preference

The token only states how your content may be used, so crawling, search indexing and each company's own robots.txt rules keep working as before. It sits next to your existing rules rather than replacing them, which matters if you expect it to stop bots or clean up old training data.

Crawlers can still fetch your pages

The draft says the token is not a crawl directive, so blocking fetches still requires a group for each crawler's own name. Microsoft makes the same point: the new settings govern how content is used, not whether a crawler can reach it. Your search listings are not affected.

The draft does allow one exception. A crawler that fetches pages only for AI training may skip pages that have opted out, so you might see fewer visits from such bots, but the draft does not require it.

GPTBot and Google-Extended rules still apply

OpenAI and Google both say their company-specific controls stay in force, and a company may treat its own token as the more specific rule when the two disagree. OpenAI says the new signal will not replace the robots.txt controls it already offers. Google says its existing controls will keep working alongside the joint signal.

The draft says specific rules should win over general ones and lets companies treat their own tokens as more specific. If your GPTBot group allows training while the shared group disallows it, a company that treats its own token as more specific would follow the GPTBot rule. Keep the two pointing the same way.

Content collected earlier is not covered

The preference that counts is the one in the robots.txt file at the moment a page was fetched, and a private licensing deal can override it. Adding the group today says nothing about copies a provider gathered last year.

The draft also lets private agreements take precedence. A publisher with a content licensing deal should read that contract, not the robots.txt file, to know how its pages may be used.

How do Google and Microsoft read the signal?

Google will apply the shared opt-out to both Gemini training and grounding, while Microsoft applies it to training only and handles answers with a separate setting. That difference decides whether opting out also removes your pages from AI answers, so check which of the two results you want before you copy a template.

Effect of a shared training opt-out at each company

Google

No Gemini training and no Gemini grounding, unless Google-Extended is set

Microsoft

No training. Grounding in Copilot answers is set per page

Google treats it as a Google-Extended block

If your robots.txt opts out with the shared token and says nothing about Google-Extended, Google plans to act as if you had disallowed Google-Extended, which also keeps your pages out of Gemini grounding. Grounding means using retrieved web content, such as search results, to support a model's answer so it is more factual. Google-Extended has controlled Gemini training and grounding since 2023.

Google-Extended has no effect on Google Search: it does not change whether a page is indexed or how it ranks. What you lose is the chance to appear in Gemini Apps answers, which Google offers only to sites that allow Google-Extended. You cannot split training from grounding at Google.

Microsoft keeps grounding separate

Microsoft lets you refuse training but still allow your pages to inform Copilot answers, because grounding has its own page-level Allow, Disallow and DisallowExtended values. A page-level training preference can also override robots.txt for that page.

Microsoft says Disallow and DisallowExtended mean less brand visibility in AI experiences but no change to Bing search eligibility. The older NoArchive and NoCache tags stop counting as training preferences after March 1, 2027.

How to opt out of AI training today

Which bots to name for each goal

Training only

Disallow GPTBot, Google-Extended and ClaudeBot

No AI search either

Also disallow OAI-SearchBot and Claude-SearchBot

Stay in Gemini

Write Allow for Google-Extended, which also permits Gemini training

Naming GPTBot, Google-Extended and ClaudeBot works now, and adding the shared group in the same file means it starts working as each company switches the signal on. None of the three companies reads the shared token yet and Anthropic is still rolling out support, so the named groups are what protect your content in the meantime.

Block the training crawlers by name

GPTBot, Google-Extended and ClaudeBot are the training crawlers for OpenAI, Google and Anthropic, and a "Disallow: /" under each one opts you out today. Anthropic says restricting ClaudeBot signals that a site's future material should be left out of its training datasets.

User-Agent: GPTBot
Disallow: /

User-Agent: Google-Extended
Disallow: /

User-Agent: ClaudeBot
Disallow: /

If you want your pages to keep showing up in Gemini answers, change the Google-Extended line to "Allow: /" rather than deleting the group. Once Google supports the shared token, a site that leaves Google-Extended out will have its shared opt-out read as a Google-Extended block.

Leave search crawlers allowed

Blocking OAI-SearchBot or Claude-SearchBot removes your site from ChatGPT search answers and can lower its visibility in Claude search, so list only the training crawlers. OpenAI says each of its settings is independent, so you can stay in ChatGPT search while keeping GPTBot out.

A blanket "User-agent: *" block catches these search bots too. OpenAI also notes that a robots.txt change can take about 24 hours to show up in ChatGPT search.

Add the shared group to the same file

Parsers that do not support the token skip its group, so adding it now costs nothing and lets it take effect on each company's start date. Put it after the named groups in the robots.txt file at the root of your domain. If your site has no robots.txt yet, create a plain text file named robots.txt with the lines below and upload it to the top level of the domain, for example example.com/robots.txt.

# Training crawlers (works today)
User-Agent: GPTBot
Disallow: /

User-Agent: Google-Extended
Disallow: /

User-Agent: ClaudeBot
Disallow: /

# Shared training signal (works as each company adopts it)
User-Agent: __usage_ai-training
Disallow: /