By Roman Aglaganov
Website owners who do not want their work used to train AI have had to opt out company by company, by blocking each crawler by name in their robots.txt file, the file that tells automated visitors what they may fetch. This week OpenAI, Google, Microsoft and Anthropic said they would support one shared rule instead.
How it works
The proposal, an Internet-Draft submitted to the Internet Engineering Task Force (IETF) by Kevin Kelley, a standards lead at Anthropic, adds a special name to the “User-Agent” line of robots.txt: “__usage_ai-training”. It is not a real crawler. A group under that name states a preference about AI training rather than about crawling. For example, this tells AI developers not to train on anything except a press-kit folder:
User-Agent: __usage_ai-training, then Disallow: / and, on the next line, Allow: /press-kit/.
Because it uses the existing robots.txt grammar, older software simply ignores it, and normal crawling rules keep working as before. If a site has no such group, no preference has been expressed. The draft also allows private agreements, such as licensing deals, to override what a site states.
Who has signed up
According to The Verge, Anthropic has already adopted the signal, Google and Microsoft plan to support it next year, and OpenAI says it will do so in the “coming months”. Meta’s communications head, Andy Stone, welcomed a common signal but did not say whether Meta will honour it.
Microsoft went further. From 1 February 2027 its crawlers will read the shared training preference and a separate, Microsoft-specific “grounding” preference, which controls whether a page can be used to inform AI answers in products such as Copilot. Microsoft says neither choice affects whether a site appears in Bing’s ordinary results, and that it will stop treating its older NoArchive and NoCache tags as AI signals after 1 March 2027. It also notes that opting out of grounding reduces a brand’s visibility in AI answers, and that “neither choice guarantees inclusion, traffic, or payment.”
Not a standard yet
The document is an individual draft, not an IETF standard, and it expires on 3 May 2027 unless it is updated. The IETF’s AI Preferences working group is still working towards a final version. It is also a preference, not a lock: it relies on companies choosing to honour it, and Microsoft says its own preferences apply to future training.
Disclosure: AI Pulse is written with AI tools, including Claude, which is made by Anthropic.
