AI Training Opt-Out: Google, Microsoft, OpenAI and Anthropic Align

- Major AI players are adopting a shared robots.txt policy token (__usage_ai-training) to simplify AI training opt-outs.
- The initiative, proposed by Anthropic via the IETF, removes the need for publishers to set individual rules for every AI crawler.
- Implementation timelines vary, with OpenAI aiming for the coming months and Google and Microsoft targeting 2027.
- The standard focuses on training preferences and does not establish licensing or payment agreements between publishers and AI firms.
The friction between content creators and generative AI developers has reached a critical juncture. For years, publishers have had to play a game of digital whack-a-mole, updating their robots.txt files with specific directives for every new AI crawler that emerged. Now, a coalition of the industry's most powerful actors—Google, Microsoft, OpenAI, and Anthropic—has announced they will honor a shared AI training opt-out signal.
This move marks a shift toward a more standardized approach to web scraping. Instead of managing a fragmented list of User-Agents, website owners can now use a single policy token to signal their preferences regarding how their data is used to modify the learned parameters of generative AI models.
The mechanics of the __usage_ai-training token
The proposed solution originates from an Internet Draft submitted to the Internet Engineering Task Force (IETF) by Kevin Kelley, a technical lead for standards at Anthropic. The mechanism does not introduce a brand-new directive to the robots.txt protocol but rather utilizes a policy token: __usage_ai-training.
Under this framework, a publisher can define a rule group specifically for this token. For example, a site could disallow AI training for its entire domain while maintaining an exception for a specific folder, such as a press kit. This allows businesses to protect their proprietary intellectual property while still allowing public-facing promotional materials to be ingested by AI models.
It is important to note that this token specifically addresses AI training—the process of using content to adjust a model's internal weights. It is distinct from non-generative tasks like simple classification or the traditional indexing used for search engine results.
Divergent implementation paths for the tech giants
While the goal of a shared signal is unified, the execution varies across the board. The companies are not moving in lockstep, and their internal interpretations of the token differ based on their existing ecosystems.
Google has indicated that it will support the signal starting in early 2027. In the absence of a specific Google-Extended directive, Google will treat the shared opt-out as a block on Google-Extended. This control extends to grounding within Gemini Apps, meaning content opted out via the token will not be used to provide real-time context for AI responses.
Microsoft is taking a different route, planning to honor the new preferences starting February 1, 2027. The company intends to replace its existing NoArchive and NoCache tags with these new preferences. Furthermore, Microsoft is introducing a separate page-level setting for grounding. This allows publishers to choose whether their live pages can be used by Microsoft's own products, such as Copilot, or if they should be blocked from grounding services that Microsoft sells to third-party companies.
OpenAI is the most aggressive on the timeline, stating its intention to implement the signal in the coming months, though it clarified that existing controls like GPTBot will remain active.
The boundary between preference and legality
Despite the collaborative nature of this announcement, the shared token is not a legal instrument. The IETF draft explicitly states that the token conveys a preference, not a mandatory order regarding crawling. This distinction is crucial for business owners to understand: the signal is a technical request for cooperation, not a binding contract.
The draft is an individual submission; the IETF's datatracker confirms that the IETF has not endorsed it and it has no formal standing in the standards process.
Furthermore, Microsoft has been explicit that these new preferences do not establish any licensing or payment arrangements. This means that opting out—or opting in—does not trigger any financial compensation. Publishers seeking monetization for their data will still need to negotiate private agreements, which the companies have noted can override the robots.txt signal.
Industry reactions and the broader scraping war
The move comes as publishers increasingly struggle with the volume of AI crawlers. The industry has seen various attempts to regain control, from Cloudflare blocking AI crawlers by default to the Really Simple Licensing (RSL) standard, which attempts to charge crawlers for access.
The support for this standard extends beyond the primary four. Meta's communications head, Andy Stone, has signaled support for the common signal, describing it as an important step for the industry, although Meta has not yet committed to a specific implementation date or method.
For the average entrepreneur, this reduces the technical overhead of protecting a website. Instead of monitoring the documentation of every new AI startup, a single line of code can now communicate a "do not train" message to the dominant players in the field.
Comparing the strategic focus of the leaders
To understand why these companies are approaching the opt-out differently, it helps to look at their core business models. While both are American multinationals, their primary drivers differ significantly.
Google's empire is built on organizing the world's information and making it accessible via internet-based services. For Google, the balance between blocking AI training and maintaining search visibility is delicate. If a site blocks everything, it disappears from search, which hurts Google's primary product. Hence, the integration with Google-Extended is designed to keep the search index healthy while respecting AI boundaries.
Microsoft, conversely, has a deeper history in computer software and enterprise productivity. With the integration of Copilot across Microsoft 365 and Windows, Microsoft is more focused on how AI enhances the productivity of the end-user. Their focus on grounding—the ability of an AI to pull from a live web page—reflects their goal of making AI a real-time assistant for business professionals.
Global implications for international businesses
For entrepreneurs and enterprises in the USA, UK, and global markets, this development simplifies the technical side of data governance but leaves the legal side open. In the United States, where fair use doctrines are often cited by AI companies to justify scraping, a technical opt-out provides a clear signal of non-consent that could be relevant in future litigation.
In the UK, where the regulatory environment is attempting to balance AI innovation with copyright protection, this standard provides a mechanism for publishers to exercise control without needing a massive legal team to send cease-and-desist letters to every AI lab.
While the EU AI Act introduces stringent transparency requirements for generative AI, the shared robots.txt token is a market-led solution rather than a regulatory one. For global businesses, the takeaway is clear: the era of "silent scraping" is ending. Companies now have a standardized way to tell the world's largest AI models to stay away from their proprietary data, but they must still pursue separate legal and commercial channels if they wish to be paid for that data.
FAQ
Does the new robots.txt token stop my site from appearing in Google Search?
No. The token is specifically for AI training (modifying model parameters). It is designed to be separate from search indexing, meaning you can opt out of AI training while still being discoverable in search results.
Will I get paid if I allow AI companies to train on my data using this token?
No. Microsoft and other partners have explicitly stated that these preferences do not establish licensing or payment arrangements.
When will this be fully active across all platforms?
Timelines vary. OpenAI aims for the coming months, while Google and Microsoft are targeting 2027.
What happens if I have a private contract with an AI company but also use the opt-out token?
Private agreements generally override the robots.txt signal.
Sources: Searchenginejournal, Geeksforgeeks, Forms ·
Scrivila qui: Susanna, l assistente AI di glacom, ti risponde via email con un approfondimento gratuito.
Nessuna consulenza personalizzata (finanziaria, legale o medica): solo informazione e fonti. Email usata solo per rispondere.
oppure scrivile su: WhatsApp · Telegram · SimpleX · Delta Chat · Email






