# OpenAI, Google, Microsoft and Anthropic back one robots.txt rule for saying no to AI training

URL: https://ai-pulse.aglaganova.com/blog/openai-google-microsoft-and-anthropic-back-one-robots-txt-rule-for-saying-no-to-ai-training
Published: 2026-10-31
Author: Roman Aglaganov, Full-stack developer
Category: Policy
Region: Global
Tags: copyright, search

> A draft web standard lets a site state a single AI-training preference that every company honouring it will read, instead of blocking each crawler by name. Anthropic already follows it; Google and Microsoft plan to next year.

![Illustration: a tall black text-file outline with grey horizontal lines, one line highlighted in red, and several small grey arrows turning away from it](https://ai-pulse.aglaganova.com/api/media/file/ai-training-signal-1.png)

Website owners who do not want their work used to train AI have had to opt out company by company, by blocking each crawler by name in their robots.txt file, the file that tells automated visitors what they may fetch. This week OpenAI, Google, Microsoft and Anthropic said they would support one shared rule instead.

## How it works

The proposal, an Internet-Draft submitted to the Internet Engineering Task Force (IETF) by Kevin Kelley, a standards lead at Anthropic, adds a special name to the “User-Agent” line of robots.txt: “__usage_ai-training”. It is not a real crawler. A group under that name states a preference about AI training rather than about crawling. For example, this tells AI developers not to train on anything except a press-kit folder:

User-Agent: __usage_ai-training, then Disallow: / and, on the next line, Allow: /press-kit/.

Because it uses the existing robots.txt grammar, older software simply ignores it, and normal crawling rules keep working as before. If a site has no such group, no preference has been expressed. The draft also allows private agreements, such as licensing deals, to override what a site states.

## Who has signed up

According to The Verge, Anthropic has already adopted the signal, Google and Microsoft plan to support it next year, and OpenAI says it will do so in the “coming months”. Meta’s communications head, Andy Stone, welcomed a common signal but did not say whether Meta will honour it.

Microsoft went further. From 1 February 2027 its crawlers will read the shared training preference and a separate, Microsoft-specific “grounding” preference, which controls whether a page can be used to inform AI answers in products such as Copilot. Microsoft says neither choice affects whether a site appears in Bing’s ordinary results, and that it will stop treating its older NoArchive and NoCache tags as AI signals after 1 March 2027. It also notes that opting out of grounding reduces a brand’s visibility in AI answers, and that “neither choice guarantees inclusion, traffic, or payment.”

## Not a standard yet

The document is an individual draft, not an IETF standard, and it expires on 3 May 2027 unless it is updated. The IETF’s AI Preferences working group is still working towards a final version. It is also a preference, not a lock: it relies on companies choosing to honour it, and Microsoft says its own preferences apply to future training.

Disclosure: AI Pulse is written with AI tools, including Claude, which is made by Anthropic.

## Sources

- [The Verge: The biggest names in AI are backing a new way to opt out of scraping](https://www.theverge.com/tech/1023763/openai-google-microsoft-ai-scraping-standard-opt-out)
- [IETF Datatracker: Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00)](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/)
- [Bing Webmaster Blog: Your content, your choice: clearer preferences for training and grounding](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/)