Google's Gary Illyes explains that robots.txt governs web crawling access, not how AI models use data after collection. Learn the implications for site owners.

For years, the robots.txt file has served as the digital handshake between website owners and automated crawlers. It is a simple text file that tells bots which parts of a site are off-limits. However, the explosion of generative AI has blurred the lines between accessing a page for search indexing and harvesting data for model training. Google has now stepped in to clarify a critical distinction that every business owner and digital strategist needs to understand.

In a recent episode of the YouTube series The Non-Canonical Cabinet, Google Search Relations experts Gary Illyes and Cherry Prommawin addressed a growing misconception: the belief that robots.txt can be used to allow a bot to crawl a site for search visibility while simultaneously forbidding that same bot from using the content for AI training. According to Illyes, this is not how the current internet standard works.

The fundamental gap between crawling and training

The core of the issue lies in the technical definition of the protocol. As defined by RFC 9309, robots.txt is strictly an access preference protocol. Its sole purpose is to govern which URLs a crawler may request. Once a bot has successfully crawled a page, the robots.txt file ceases to have any authority over what happens to that data.

Illyes explicitly stated that the protocol does not dictate what a system does with the data after it has already been collected. This means that if a site owner allows a crawler to enter, they cannot use the same file to tell the AI company not to train their models on that content. To prevent an AI bot from using your data, you must prevent it from accessing the site entirely by targeting its specific user-agent token. If the bot cannot crawl the page, it cannot ingest the data for training.

Why custom rules and emojis fail

Many site owners, frustrated by the lack of granular AI controls, have attempted to implement homemade directives or use visual cues like the ⛔ emoji to signal a no-entry zone for AI. Google's guidance is blunt: these efforts are useless. The standard robots.txt parser only recognizes four elements: User-Agent, Allow, Disallow, and Sitemap.

Any line that does not follow this syntax—including fictional directives like No-AI-Chicken-Training or ASCII art—is simply ignored by the parser. The system does not see a request for privacy; it sees an invalid line of text and silently drops it. For those managing global sites, Cherry Prommawin also highlighted a common technical pitfall regarding internationalization. She recommended using percent-encoding for non-English rule paths to ensure that crawlers other than Googlebot interpret the paths correctly.

The unique role of Google-Extended

While the general RFC 9309 standard does not cover usage, Google has introduced a proprietary mechanism called Google-Extended. It is important to note that Google-Extended is not a user-agent in the traditional sense; it is a token used in a control capacity. It allows site owners to manage whether content crawled by Google is used to train future Gemini models or used for grounding in Gemini Apps and Vertex AI.

This creates a two-tier system. While the standard robots.txt handles the crawling access, Google-Extended acts as a specific usage control based on Google's own internal documentation. Crucially, using Google-Extended does not affect a website's inclusion in Google Search results, nor does it serve as a ranking signal.

Moving toward a new global standard

The industry recognizes that the current gap between access and usage is a problem. To solve this, Gary Illyes, alongside Mozilla's Martin Thomson, has co-authored an IETF Internet-Draft. This proposal seeks to update RFC 9309 by introducing a formal Content-Usage rule.

If approved and adopted, this would provide a standardized way for website owners to state their usage preferences directly within the robots.txt file. Instead of relying on proprietary tokens like Google-Extended or hoping that a bot respects a non-standard line of text, publishers could use a globally recognized command to forbid AI training while still permitting search indexing. Until this draft becomes a reality, however, the burden of protection remains on the site owner to identify and block specific AI user-agents.

Technical best practices for site administrators

To ensure that a website is actually protected from unwanted AI harvesting, administrators should move away from intuitive but ineffective methods. The focus must shift toward strict adherence to the RFC 9309 standard. This involves a rigorous approach to identifying the exact user-agent tokens of the AI bots they wish to exclude.

Robots.txt isn't a security mechanism. It requests that compliant crawlers refrain from accessing specified paths; it doesn't prevent access by unauthorized clients or guarantee that every bot will comply.

Validation is key. Site owners should not assume a rule is working simply because it is human-readable. Testing actual URL patterns and validating syntax is the only way to confirm that a bot is being successfully blocked. For those operating in multiple languages, the recommendation to use percent-encoding is not optional if they want consistent behavior across different crawler implementations.

Strategic implications for US and UK businesses

For entrepreneurs and enterprises in the USA and UK, this clarification underscores a significant legal and technical vulnerability. In these jurisdictions, the concept of AI training via crawling often sits in a grey area of copyright and fair use. Relying on a robots.txt file to protect intellectual property is a technical misunderstanding; as Google has clarified, the file is a request for access, not a legal contract or a data-use agreement.

US-based companies should be aware that since robots.txt is not a security mechanism, it does not constitute a technical barrier to data scraping. For businesses with high-value proprietary data, implementing server-side blocks or authentication layers is more effective than relying on a text file. In the UK, where data protection and intellectual property rights are strictly monitored, the distinction between crawling and training is vital for any potential litigation regarding unauthorized data ingestion.

The takeaway for the global market is clear: the current tools for AI governance are fragmented. Until the IETF draft introduces a standardized Content-Usage rule, businesses must be proactive. This means auditing their robots.txt files to remove useless emojis or custom directives and replacing them with explicit user-agent blocks for any AI crawler they do not wish to have on their servers.

FAQ

---

Sources: Searchenginejournal, Semasocial, Ppc ·