Block AI Crawlers in robots.txt, Keep AI Search (2026)

How to keep your content out of AI training sets in 2026 without disappearing from ChatGPT, Gemini, Claude, Perplexity and Google's AI answers

On October 30, 2026, OpenAI, Google and Microsoft each committed to reading one shared robots.txt line that opts a site out of AI training. OpenAI said it will honor a shared training signal that draws on the IETF's AI Preferences work and will implement it "in the coming months" - OpenAI. Microsoft published the exact syntax the same day and named February 1, 2027 as the date its crawling starts acting on it - Microsoft Bing. Google added a note to its crawler documentation saying it intends to support the same token in early 2027 - Google. The token itself, __usage_ai-training, comes from an Internet-Draft written by Kevin Kelley of Anthropic and posted that same day - IETF.

But one line in robots.txt does not decide whether AI search keeps sending you visitors, and most "block AI" setups get this wrong. The labs run separate machines for separate jobs: one crawler collects pages for training, another indexes pages so an assistant can cite and link them, and a third fetches a page live because a person asked about it. Block the wrong one and you vanish from the answers that now carry a growing share of discovery. Allow everything and your content trains models that may answer your readers' questions without ever sending them to you. The tension is real, and it is measurable: on Cloudflare's network, training accounted for 42.6% of AI crawler requests in the 28 days to November 4, 2026, while pure search crawling was 12.1% - Cloudflare Radar.

This guide breaks down exactly what changed on October 30, which crawler does which job at every major AI company, the copy-paste robots.txt that blocks training while keeping search, what to do about the crawlers that ignore the file, how Google's AI Overviews fit in (they are the hard case), and the economic question underneath all of it: whether blocking training is even the right move for your site. We also report what happened when Founden, which builds and runs autonomous software companies and hosts their sites, shipped exactly the configuration in this article's title to every company it built from June 2026, measured the result, and reversed the default on November 1.

Contents

1. What Changed on October 30, 2026: One Training Opt-Out for Every Lab
2. Training, Search and Agents: The Three Jobs Behind Every AI Crawler
3. The Economics: What Each Crawler Takes and What It Sends Back
4. Who Blocks What Today: What the Top Sites' robots.txt Files Show
5. The Crawler Reference: Every Token That Matters in November 2026
6. The robots.txt Itself: Templates That Block Training and Keep Search
7. Beyond robots.txt: Enforcement When Crawlers Ignore the File
8. Google Is the Hard Case: Training, AI Overviews and Search
9. What Founden Learned Blocking Training Crawlers, Then Welcoming Them
10. A Decision Framework: Should You Block AI Training at All?
11. The Legal Layer: What an Opt-Out Is Worth in 2026
12. Monitoring: Prove Your Policy Works
13. Future Outlook: Grounding, Signed Agents and Paid Crawling

The Control Methods, Scored

There are now at least eight distinct ways to tell AI companies not to train on your content, and they differ on the four things a site owner actually cares about. The table below scores each one on those four criteria, using what is documented and live as of November 4, 2026, not what has been promised. Every cell carries the score and the fact behind it.

How the criteria were chosen and weighted. Training coverage today (30%) asks how much training-bound crawling the method actually stops or opts out of right now, because a signal nobody honors yet protects nothing this quarter. Search traffic kept (30%) carries equal weight because the whole point of this guide is not paying for a training opt-out with lost discovery. Enforcement (20%) measures whether the method still works against a crawler that ignores the rules. Effort and cost (20%) reflects what a small team can realistically set up and keep current. Rows 6 and 7 tie at 5.3 and are ordered alphabetically.

The ranking hides one important point: the methods stack. The strongest setup in November 2026 is rows 2 and 4 together in your robots.txt (the per-crawler groups that work today plus the shared group that will work across labs next year), backed by row 1 or row 3 at the network edge for the crawlers that ignore the file. Sections 6 and 7 build exactly that.

1. What Changed on October 30, 2026: One Training Opt-Out for Every Lab

Until October 30, opting out of AI training meant learning a different robots.txt token for every company, and the list kept changing. OpenAI had GPTBot, Anthropic had ClaudeBot, Google had Google-Extended, Apple had Applebot-Extended, Meta had meta-externalagent, and Microsoft had no robots.txt training token at all, reading the old noarchive and nocache meta tags instead. Every new lab added a new name, and every renamed crawler silently broke somebody's opt-out. A site owner who set up a careful block list in 2024 was, by late 2026, still blocking tokens that their owners no longer document while missing the ones that now matter.

The October 30 announcements replace that treadmill with one shared name for one purpose. The design is deliberately boring. The Kelley draft adds no new directives and no new grammar to robots.txt: it uses an ordinary group whose User-agent line names a "policy token" instead of a crawler, and the existing Allow and Disallow rules, longest-match precedence and group structure of the robots.txt standard stay exactly as they are - IETF. A disallow under that token means "I prefer you do not use this for AI training," and an allow means the opposite.

1.1 The syntax, and the three rules that trip people up

The whole opt-out is two lines. The draft's own example opts an entire site out of training except one folder that the owner wants models to learn from, which is a pattern worth copying for press kits and public documentation:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Three details in the draft decide whether this works for you. First, a User-agent: * group is explicitly not treated as matching the policy token, so a blanket rule says nothing about training, and a file without a __usage_ai-training group expresses no preference at all. Second, the token "is not a directive for crawling behavior": it governs what may be done with a page, not whether a crawler may fetch it, so you still need crawler-specific rules if you want to stop the requests themselves. Third, the preference that applies is the one in the robots.txt file in effect when the page was retrieved, and the draft notes that private agreements between a site and a lab may override it.

• Wildcard does not count: User-agent: * never expresses a training preference
• No group, no preference: silence is not an opt-out
• Use, not crawl: the token limits training, not fetching
• Point in time: the robots.txt at retrieval time decides

Those four properties are the reason the signal can be adopted quickly without breaking anything. Because it is a separate group, every existing crawler keeps reading its own rules exactly as before, and legacy parsers simply treat __usage_ai-training as an unknown user agent and ignore it. Because it governs use rather than crawling, a search crawler can keep fetching your pages for its index while the same company respects your preference about training. The cost of that cleanliness is that the token stops nothing on its own, which is why the scoring table puts its enforcement at 2 out of 10.

1.2 Who committed to what, and when it takes effect

The commitments arrived the same day but on different timelines, and none of them is live yet. Microsoft's post is the most concrete: it gives whole-site and path-level syntax, adds a page-level <meta name="Bingbot" content="ai-training:disallow"> option, and states that the new preferences start taking effect as it crawls from February 1, 2027, with the old NoArchive and NoCache tags no longer read as training or grounding preferences after March 1, 2027 - Microsoft Bing. Google's crawler changelog records an October 30 update that mentions "future AI training control support" - Google. OpenAI said it will share implementation details when it ships, and that its existing robots.txt controls stay in force in the meantime.

The practical reading of this table is simple: add the shared group now, because it costs nothing and starts working lab by lab as support ships, but do not remove a single per-crawler rule yet. Until a provider confirms it reads the shared token, its own token is the only opt-out it honors. Microsoft is the exception that needs action before March 1, 2027, because sites that relied on noarchive to keep content out of Bing's AI answers will lose that protection unless they publish the new page-level grounding tag.

1.3 Where the standard actually stands

It is worth being precise about the status, because "IETF standard" is already being used loosely in coverage. The AI Preferences working group has two documents on the standards track: a vocabulary for expressing AI usage preferences, at revision 08 from September 13, 2026, and a document for attaching those preferences to content in HTTP, at revision 05 from August 18, 2026 - IETF Datatracker. Neither is an RFC yet. The Kelley draft is a separate individual submission that borrows the vocabulary's definition of AI training ("using an asset to modify the learned parameters of a generative AI model") and turns it into a robots.txt token.

The attach draft, written by Google's Gary Illyes and Mozilla's Martin Thomson, defines a broader Content-Usage rule and HTTP header for the full vocabulary - IETF. The October 30 move effectively shipped the narrowest, least controversial piece first: one category, training, in a syntax every robots.txt parser already understands. That matters for how you plan. Expect more categories (grounding, search, summaries) to arrive the same way over the next year, and expect the per-company tokens to linger for years alongside them, the way noarchive lingered at Bing.

1.4 Why this landed now

The timing is not an accident. The labs that need the open web for training are now also the companies building the surfaces where people discover products, and their own business depends on sites not blocking them wholesale. Anthropic's S-1 filing in late October, which we unpacked in our guide to what the S-1 means for Claude builders, made the scale of that business public. Google opened its Connected Apps Platform preview two days earlier, putting products directly inside Search and the Gemini app, as we covered in our Google Connected Apps guide. A credible, uniform training opt-out lowers the pressure on site owners to block everything, which protects the search and agent crawling those surfaces depend on.

Regulation pushed in the same direction. The EU's General-Purpose AI Code of Practice commits signatories to honoring robots.txt and, beyond it, other appropriate machine-readable protocols "adopted by international or European standardisation organisations" - Paul, Weiss. A training token that comes out of the IETF fits that description far better than a dozen proprietary names. Section 11 returns to what that means legally. For now, the takeaway for builders is that the shared token is likely to become the one training opt-out that regulators, labs and CDNs all recognize, so it belongs in your file today.

2. Training, Search and Agents: The Three Jobs Behind Every AI Crawler

Most bad robots.txt decisions start from treating "AI bots" as one category. They are not. Every request an AI company sends to your site serves one of three jobs, and each job has a different relationship to your traffic. Training crawlers collect pages to change a model's learned parameters; nothing about that request sends you a visitor, now or later, except indirectly through what the model remembers. Search and indexing crawlers build an index the assistant retrieves from when it answers a question, and that index is where citations and links come from. User-triggered fetchers load a specific page in real time because a person pasted a link, asked an agent to check something, or asked a question that needed fresh information.

Cloudflare, which sees a large share of web traffic, now classifies crawlers along exactly these lines. Over the 28 days to November 4, 2026, training was 42.6% of AI crawler requests, mixed-purpose crawlers (one bot doing several jobs) 39.5%, search 12.1%, user actions 4.3% and undeclared purposes 1.5% - Cloudflare Radar. Cloudflare's own July 2026 report put training at 52% of crawler requests as of June 2026, up from 22% in spring 2025 - Cloudflare. The chart below shows the monthly trend from the same Radar series.

Two movements in that chart matter for your decision. Training climbed to a peak of more than half of AI crawling in June and has eased since, while dedicated search crawling grew roughly fivefold from the start of the year to October. The mixed-purpose share fell as companies split their crawlers by job. That split is precisely what makes "block training, keep search" possible at all: you can only separate two uses when they arrive under two names.

2.1 Crawling is access; training is use

There is a conceptual distinction underneath the three jobs that explains why the shared token is built the way it is. The robots.txt standard, RFC 9309, controls whether a crawler may fetch a URL, and it says so plainly: "These rules are not a form of access authorization" - RFC Editor. It was never designed to say what may be done with content after it is fetched. Tokens like Google-Extended, Applebot-Extended and __usage_ai-training stretch the file into a second role: they do not stop any crawler, they express a preference about how content fetched by another crawler may be used.

Google's Search Central team made exactly this point in an October 21 episode of its Non-Canonical Cabinet series, and it is the clearest short explanation of the access-versus-use problem available. In the section starting at 07:06, Gary Illyes and Cherry Prommawin explain that a disallow rule means "do not enter the room" and does not dictate what a system does with data it already crawled, and that blocking a specific AI crawler requires targeting its own token explicitly, since invented rule names are simply discarded by parsers - PPC Land.

The video's closing advice, "keep your robots.txt beautifully boring," is good engineering advice here too. Every rule you add should name a documented token and do one job. Creative syntax, made-up directives and emoji (the video tests all of them) are ignored at best. The deeper lesson is that use preferences only work when the company doing the training reads the token you wrote, which is why the October 30 commitments matter more than any clever robots.txt trick.

2.2 Why user-triggered fetchers ignore your file

The third job, real-time fetching on a person's behalf, is where robots.txt stops applying by design. OpenAI states that because ChatGPT-User actions are initiated by a user, "robots.txt rules may not apply" - OpenAI. Perplexity says its Perplexity-User fetcher "generally ignores robots.txt rules" - Perplexity. Google applies the same logic to its user-triggered fetchers, including the Google-Agent user agent it added in March 2026 for agents that act on user requests - Google.

The reasoning is that a person asking an assistant to read a page is closer to a person opening it in a browser than to a crawler. Whether you accept that framing or not, it has a practical consequence: these fetchers are the closest thing to a visitor in the AI stack, and blocking them blocks the person behind them. If you sell to people who increasingly research with agents (we covered that shift in our guide to selling to AI agents), a user-triggered fetcher is a customer showing up through a different door. The control that makes sense for them is rate limiting and abuse detection, not a robots.txt line they were never meant to obey.

With the three jobs separated, the rest of the guide follows naturally. Training crawlers are the ones the shared token and the per-company training tokens target. Search crawlers are the ones you almost certainly want to keep. User-triggered fetchers are effectively visitors. The next question is what each of those groups actually gives back, because that is what should decide your policy, not a vague sense that "AI is taking our content."

3. The Economics: What Each Crawler Takes and What It Sends Back

Every crawler request has a cost and a possible return. The cost is bandwidth, compute and, on usage-billed hosting, real money per request. The return is a visitor, a citation, a mention in an answer, or a licensing fee. A search engine that crawls your page and sends a reader back is a trade most sites are glad to make. A crawler that takes thousands of pages and sends nobody is pure cost. Measuring that ratio, crawl to referral, is the closest thing the industry has to a price list for AI crawling.

Cloudflare publishes exactly that ratio from its network. In October 2026, Perplexity's crawlers fetched about 4,020 pages for every visit Perplexity referred back, OpenAI about 800, Anthropic about 490, Microsoft about 40, and Google about 3.8 - Cloudflare Radar. Mistral's figure for the same month was over 107,000, though its small referral counts make that number noisy. The chart below shows the October values for the five largest referrers.

These ratios move fast, which is itself an argument against setting a policy once and forgetting it. Anthropic's ratio was about 47,800 to 1 in January 2026 and fell to about 490 to 1 by October as Claude's search and referral traffic grew. Perplexity moved the other way, from roughly 110 to 1 in January to 4,020 to 1 in October. OpenAI sat between about 250 and 1,300 to 1 all year. The line chart below tracks OpenAI and Perplexity, the two platforms whose curves crossed.

3.1 Why the ratio misleads if you read it alone

A crawl-to-refer ratio treats every crawl as a cost to be repaid in clicks. That works for search crawling, where the deal has always been "index my page, send me readers." It misreads training, which never promised a click. When a model learns that your company exists, what it does and who it is for, the return arrives later and without a referrer: the assistant names you in an answer, the user remembers the name, and they search for it or type your address. That visit shows up in your analytics as direct or branded search traffic, not as an AI referral, so it never enters the ratio.

The size of the measurable AI referral channel is still small enough that this invisible part matters. Conductor's benchmark across 13,770 domains found AI referrals were 1.08% of total website traffic between May and September 2025 - Scalevise. Clicks on links inside AI answers are scarce even in Google: Pew found users clicked a traditional result on 8% of visits when an AI summary appeared versus 15% without one, and clicked a link inside the summary itself on about 1% of visits - Pew Research Center. TollBit measured click-through from AI apps falling from 0.8% in Q2 2025 to 0.27% in Q4 2025 - Digiday.

The referral mix is also shifting under everyone's feet. Statcounter, which counts the visits each chatbot sends to websites, showed ChatGPT at about 80% of chatbot referrals through September 2026, then 51.58% in October, with Gemini at 28.86% and Perplexity at 15.04% - Statcounter. Statcounter's own releases describe the metric as referral share rather than usage share - Statcounter. A policy tuned to one assistant's behavior can be wrong within a month.

3.2 What blocking actually cost the publishers who did it

The best causal evidence on blocking comes from news publishers, because many of them blocked OpenAI's crawlers early and at identifiable dates. Hangcheng Zhao of Rutgers and Ron Berman of Wharton used those dates in a staggered difference-in-differences design and found that blocking GenAI crawlers was associated with roughly a 7% decline in traffic within six weeks, measured with SimilarWeb data, with a similar but statistically insignificant decline in a human-only Comscore panel and effects concentrated among the largest publishers - arXiv. The estimate faded toward insignificance after about 20 weeks, and earlier versions of the same paper reported larger numbers, so treat 7% as the current best estimate rather than a law.

Even with those caveats, the direction is informative. The publishers in that study blocked crawlers that, at the time, mostly served training and mixed purposes. If blocking those cost measurable traffic, the cost of accidentally blocking a search crawler, whose whole job is sending readers, is likely higher. On the other side, Cloudflare cites outside studies finding that AI-referred visitors convert at between three and more than five times the rate of traditional search referrals - Cloudflare. Small channel, valuable visitors, and a real cost to getting the block wrong.

The first-principles summary of this section is that a crawler should be judged by what its owner gives back, in every form, not by request counts alone. Search crawlers give back clicks and citations. Training crawlers give back presence in the model's memory, which is worth a lot to a company that wants to be named and very little to a publisher whose content is the product. Crawlers whose owners have no way to send you anyone (SEO tools, data resellers, training for products your audience never uses) give back nothing. Section 10 turns that into a decision procedure; the next section shows what sites are actually doing.

4. Who Blocks What Today: What the Top Sites' robots.txt Files Show

Cloudflare Radar parses the robots.txt files of the most visited domains and counts which user agents each file disallows. On November 2, 2026, it parsed 4,399 files, and the pattern in them is clear: sites already treat training and search as different decisions. GPTBot was fully disallowed by 512 files, about 11.6%, while OpenAI's search crawler OAI-SearchBot was fully disallowed by 124, about 2.8% - Cloudflare Radar. The gap is even wider at Google, where Google-Extended was fully blocked by 417 files (9.5%) and Googlebot by just 14 (0.3%).

The pairs in the chart are GPTBot and OAI-SearchBot, ClaudeBot and Claude-SearchBot, Google-Extended and Googlebot, meta-externalagent and meta-webindexer, and Applebot-Extended and Applebot. Across all five companies, top sites block the training token three to thirty times more often than the search crawler. Cloudflare's own count across its customer base points the same way: fewer than 1% of Cloudflare sites block search bots, while 17% enable some mechanism to block training - Cloudflare.

4.1 Three things the raw counts reveal

The first is that the opposite choice is common too. In the same parse, 368 files explicitly named GPTBot in an Allow rule, and PerplexityBot was explicitly allowed by 357. Some sites are not merely tolerating AI crawlers, they are inviting them by name, which is a deliberate bet on being learned and cited. The second is that publishers are far more aggressive than the web at large: roughly 75% of the news publishers in the Zhao and Berman sample blocked an OpenAI-related crawler at some point, against about 12% of top domains overall - arXiv.

The third finding is the most actionable: block lists rot. The same Radar parse shows 436 files still disallowing anthropic-ai and 383 disallowing Claude-Web, while Anthropic's current documentation lists only ClaudeBot, Claude-User and Claude-SearchBot - Anthropic. Those files carry tokens Anthropic no longer documents. A site that copied a 2024 block list and never revisited it may be blocking names that no longer correspond to a crawler while missing the ones that do, which is exactly the maintenance problem the shared token is meant to end.

Put the three findings side by side and the picture is consistent. Training tokens are fully blocked by roughly 8% to 12% of top sites, search crawlers by roughly 0.3% to 3%, and about 8% of top files go out of their way to invite GPTBot by name. The sites that block are concentrated among publishers, the sites that invite are betting on being learned, and a sizable minority on both sides are maintaining lists that no longer match what the operators run.

The early adoption of the shared token is not yet visible in this data: __usage_ai-training did not appear among the 100 most-disallowed user agents in the November 2 parse, three days after it was defined. That will change quickly if CDNs start writing it by default. For your own file, the lesson from this section is to separate the decisions the way the leading sites already do, keep your tokens current, and accept that "block or allow" is a choice reasonable sites make in both directions. The next section is the reference you need to make it correctly.

5. The Crawler Reference: Every Token That Matters in November 2026

Every token in this section comes from the operator's own documentation as of November 2026. The table maps each company's three jobs to the names you would write in robots.txt. Where a company uses one crawler and a separate control token, the token is what goes in your file. Where a company runs no training crawler, the cell says so.

The table is the whole map, but several rows hide details that change what you should write. The subsections below walk through them company by company, with the source for each claim, because a robots.txt built from a secondhand list is how sites end up blocking the wrong crawler.

5.1 OpenAI and Anthropic: three clean names each

OpenAI runs the clearest split in the industry. GPTBot crawls content that may be used to train its foundation models, OAI-SearchBot surfaces sites in ChatGPT's search features, and sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers," though they can still appear as navigational links - OpenAI. OpenAI also notes that when a site allows both crawlers it may use one crawl for both purposes to avoid duplicate requests, and that robots.txt changes take about 24 hours to reach its systems. A fourth agent, OAI-AdsBot, only visits landing pages submitted as ChatGPT ads and does not collect training data.

Anthropic mirrors the structure. ClaudeBot collects content that could contribute to training, Claude-SearchBot improves search result quality, and Claude-User fetches pages for Claude users; Anthropic warns that blocking either of the latter two may reduce your visibility in Claude's answers - Anthropic. Two operational details are worth noting. Anthropic supports the non-standard Crawl-delay directive, which is gentler than a block if your concern is load rather than training. And it warns that blocking its IP addresses can make an opt-out fail, because it prevents the crawler from reading the robots.txt that contains your preference.

5.2 Google and Apple: one crawler, separate tokens

Google and Apple cannot split their crawlers the way OpenAI does, because the same crawler feeds their search products and their AI. Both solve it with a control token that has no crawler behind it. Google-Extended governs whether crawled content may be used for training future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI, and Google states it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" - Google. Notice what that means: disallowing Google-Extended also removes your content from Gemini app grounding, which is a form of AI search. Section 8 deals with that trade-off.

Apple's version is Applebot-Extended, which "does not crawl webpages" and only controls how data Applebot already crawled may be used, including for training the models behind Apple Intelligence - Apple. Apple states that rules for Applebot-Extended are not considered in ranking for Search, so pages that block it still appear in Spotlight, Siri and Safari results. Blocking the plain Applebot token is a different and much bigger decision, since it removes you from those search surfaces too.

5.3 Microsoft, Meta, Amazon, Perplexity and the rest

Microsoft is moving from tags to tokens. Until February 2027, Bing reads noarchive and nocache as AI preferences; after that, training follows the shared __usage_ai-training group or a page-level ai-training:disallow tag, while grounding in AI answers has its own page-level control with allow, disallow and disallowextended values (the last limits grounding to Microsoft's own products) - Microsoft Bing. Microsoft states there is no site-wide robots.txt grounding preference, and that neither control changes eligibility for traditional Bing results.

Meta documents meta-externalagent for use cases including training foundation models, meta-webindexer for improving Meta AI search results and citations, and meta-externalfetcher for fetches requested by users, which "may bypass robots.txt rules" - Meta. Amazon separates Amazonbot, whose data "may be used to train Amazon AI models," from Amzn-SearchBot, which does not crawl for generative AI training and powers search in products like Alexa; Amazon honors noarchive as "do not use for model training" and may use a robots.txt copy cached for up to 30 days - Amazon.

Perplexity says PerplexityBot exists to surface and link websites in its search results and "is not used to crawl content for AI foundation models," and it recommends allowing it - Perplexity. Common Crawl's CCBot is the one to consider carefully: Common Crawl publishes an open archive that many model builders train on, and its own page tells site owners to disallow CCBot to keep it from crawling - Common Crawl. Blocking CCBot affects future crawls only; past snapshots remain in the archive.

For everything outside this table, the community-maintained ai.robots.txt project keeps a list of AI-related crawlers with ready-made robots.txt, nginx, Apache, Caddy and HAProxy files, and has about 4,200 GitHub stars. Use it as a discovery tool, not as your policy: it lists AI crawlers "of all types, regardless of purpose," so copying it wholesale would also block search crawlers and user-triggered fetchers you probably want to keep. With the map in hand, the next section turns it into an actual file.

6. The robots.txt Itself: Templates That Block Training and Keep Search

A robots.txt that blocks training and keeps search has three parts: groups that name each company's training token, a group for the shared training token, and a default group that leaves everything else (including every search crawler) alone. Order does not matter to compliant parsers, because each crawler looks for the group that matches its own token. What matters is that you never put a search crawler's token in a disallow group by accident, and that your default group does not quietly block more than you meant.

Before copying anything, decide two things that the template cannot decide for you. First, whether to include Google-Extended, given that it also removes your content from Gemini app grounding. Second, whether to include Amazonbot, whose crawl serves both Amazon's model training and other product uses. The template below includes both, which is the conservative "no training" choice; delete a line if your answer differs.

6.1 Template A: block training, keep search (works today)

This is the file to ship in November 2026. It combines the per-crawler training tokens every lab honors today with the shared token that will start working as support lands in 2027. RFC 9309 allows a group to start with several user-agent lines, so one block covers every training crawler at once.

# AI training: opt out (per-company tokens, honored today)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: CCBot
User-agent: Bytespider
Disallow: /

# AI training: opt out (shared IETF policy token, honored as labs ship support)
User-agent: __usage_ai-training
Disallow: /

# Everyone else, including Googlebot, Bingbot, Applebot, OAI-SearchBot,
# Claude-SearchBot, PerplexityBot, meta-webindexer and Amzn-SearchBot
User-agent: *
Allow: /
Disallow: /api/
Disallow: /account/

Sitemap: https://example.com/sitemap.xml

Read the default group carefully, because it is where most mistakes hide. Everything that does not match a named group falls through to User-agent: *, so the search crawlers keep exactly the access they had before. The two disallowed paths are illustrative; keep whatever private or non-page routes your site already excludes. Do not add AI search crawlers to the training block "to be safe," and do not add a Disallow: / to the default group, which would remove you from every search engine at once.

6.2 Template B: path-level choices

Many sites want models to learn some pages and not others: public documentation and a press kit in, paid research and member content out. The robots.txt standard resolves conflicts by the most specific (longest) matching rule, so you can combine a broad allow with targeted disallows - RFC Editor. This example lets training crawlers learn the marketing site and docs while keeping premium content out of training sets.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: __usage_ai-training
Allow: /
Disallow: /research/
Disallow: /members/
Disallow: /courses/

User-agent: *
Allow: /

The Kelley draft explicitly allows policy tokens and ordinary crawler tokens in the same group, which is why __usage_ai-training can sit in this block - IETF. Path-level rules are only as good as your URL structure: if premium articles live at the same paths as free ones and are gated by a paywall script, a training crawler that fetches the page may still see the content. Put content you want excluded under its own path prefix, or serve it only to signed-in users.

6.3 Template C: generating it in Next.js

If your site runs on Next.js, generate the file from code so the list lives in one place and every deploy ships it. The App Router's metadata route accepts an array of user agents per rule, which maps directly onto multi-line groups.

// app/robots.ts
import type { MetadataRoute } from "next";

const AI_TRAINING = [
  "GPTBot", "ClaudeBot", "Google-Extended", "Applebot-Extended",
  "meta-externalagent", "Amazonbot", "CCBot", "Bytespider",
  "__usage_ai-training",
];

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: AI_TRAINING, disallow: "/" },
      { userAgent: "*", allow: "/", disallow: ["/api/", "/account/"] },
    ],
    sitemap: "https://example.com/sitemap.xml",
  };
}

Keeping the list in code has a second benefit: you can test it. A single unit test that asserts OAI-SearchBot, Claude-SearchBot, Googlebot, Bingbot and PerplexityBot are never in the training array prevents the most expensive mistake in this guide from ever reaching production. For a broader set of checks that agents can run against a site (robots.txt, sitemaps, canonical tags), our roundup of technical SEO agent skills covers the tools worth wiring into a deploy pipeline.

6.4 The mistakes that cost search traffic

The errors we see most often are not syntax errors. They are category errors: a rule that does what it says, aimed at the wrong job. A parser will happily obey a perfectly formatted file that removes you from every search engine, and nothing in your build or your logs will warn you that it happened. The damage shows up weeks later as a slow slide in organic traffic that nobody connects to a one-line change.

The reason these errors are so common is that the names look alike. Applebot and Applebot-Extended differ by one word and by an entire product surface; GPTBot and OAI-SearchBot come from the same company and do opposite jobs. Lists copied from blog posts and plugins mix them freely. The first group of mistakes removes search visibility directly.

• Blocking Googlebot to stop AI Overviews, which removes you from Google Search entirely
• Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot when the goal was training
• Blocking plain Applebot instead of Applebot-Extended, which removes you from Siri and Spotlight
• A wildcard disallow that was meant to be temporary and never came off

Each of these turns a training decision into a search decision. The fix is always the same: name the training token, never the search crawler, and use the Search Console control in Section 8 if your actual concern is Google's AI Overviews. If you inherited a robots.txt from an agency or a plugin, read every disallow group against the table in Section 5 before assuming it does what its comments claim. Search Console's robots.txt report and the coverage report will show you quickly whether Googlebot can still reach the pages that matter, and a sudden rise in pages marked as blocked is the clearest early warning you will get.

The second group of mistakes makes the opt-out silently fail rather than hurting search. They are just as common, and harder to notice, because nothing visibly breaks.

• Stale tokens like anthropic-ai and Claude-Web instead of ClaudeBot
• Missing subdomains, since each host needs its own robots.txt
• Expecting retroactive removal, when preferences apply from the next fetch onward
• Ignoring caches: about 24 hours at OpenAI, Perplexity and Meta, up to 30 days at Amazon

None of these produce an error. A stale token is simply ignored, a subdomain without its own file has no rules, and content fetched before your change stays wherever it went. That is why Section 12 ends with a review cadence: robots.txt for AI is a file you revisit when the operators change their documentation, which in 2026 happened roughly every quarter. The cheapest safeguard is to keep your list in code, next to a test that fails when a documented token is missing or a search crawler appears in a disallow group, so a stale list breaks the build instead of quietly breaking your opt-out.

6.5 Test before you ship

Testing a robots.txt takes a minute and saves weeks of lost traffic. Python's standard library includes a parser that follows the same group logic most crawlers use, so you can assert exactly which agents can fetch which paths before the file goes live.

from urllib.robotparser import RobotFileParser

rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()

for agent in ["GPTBot", "ClaudeBot", "Google-Extended", "__usage_ai-training",
              "OAI-SearchBot", "Claude-SearchBot", "Googlebot", "Bingbot", "PerplexityBot"]:
    print(f"{agent:22} can fetch /blog/: {rp.can_fetch(agent, 'https://example.com/blog/')}")

The output should read False for every training token and True for every search crawler. Run it against your production URL after each deploy, and add it to CI if your robots.txt is generated. Google's Search Console robots.txt report will show you how Googlebot itself parsed the file, which is the one parser whose interpretation you cannot afford to guess at. With the file correct, the remaining problem is the crawlers that never read it.

7. Beyond robots.txt: Enforcement When Crawlers Ignore the File

robots.txt is a request, not a lock. Compliant crawlers from the large labs follow it, and their documentation says so, but nothing technical stops a crawler that decides not to. That gap has grown as AI crawling has grown, and it matters most for the sites with the most valuable content, because those are the sites worth ignoring the rules for. If your training opt-out has real economic stakes, a robots.txt line is the start of the policy, not the whole of it.

The measured scale of non-compliance is large enough to plan for. TollBit, which monitors bot traffic for publishers, found that 30% of AI bot scrapes in Q4 2025 bypassed explicit robots.txt permissions, and that 42% of scrapes by OpenAI's ChatGPT-User reached content on sites that had blocked it - Digiday. Part of that is user-triggered fetching, which operators say robots.txt was never meant to govern. Part of it is not.

7.1 Declared crawlers, undeclared crawlers and stealth

The difference between those two parts is whether the crawler tells you who it is. A declared crawler uses its documented user agent and its published IP ranges, so you can see it, block it, and verify it. An undeclared crawler presents itself as an ordinary browser, often from rotating addresses, so neither robots.txt nor a user-agent rule touches it. In August 2025 Cloudflare reported that Perplexity, when blocked, used undeclared crawlers with browser-like user agents and IPs outside its published ranges, across tens of thousands of domains and millions of requests a day, and that it de-listed Perplexity as a verified bot - Cloudflare.

That episode is the best illustration of why robots.txt and enforcement are separate layers. A robots.txt rule only works on crawlers that want to be seen following it. A user-agent firewall rule works on declared crawlers that ignore robots.txt. Only behavioral bot detection (the kind a CDN runs across millions of sites) catches crawlers that disguise themselves. Most sites need the first two; sites whose content is genuinely valuable to scrapers need all three.

7.2 Firewall rules at the host

The cheapest enforcement layer is a firewall rule at your host or CDN that denies declared AI training crawlers before your application runs. On Vercel, the AI Bots Managed Ruleset is available on all plans and splits AI traffic into two categories you can deny independently: AI Crawlers, which collect content for training, and AI Assistants, which retrieve pages for user actions or search - Vercel. The same page documents a CLI form, which is useful if you manage several projects:

vercel firewall rules edit ai-bots --category ai-crawlers --action deny --yes
vercel firewall rules edit ai-bots --category ai-assistants --action allow --yes
vercel firewall publish --yes

A denied request also stops costing you money. Vercel states that traffic its firewall denies, challenges or rate-limits does not incur CDN request or data transfer charges - Vercel. That turns enforcement into a cost control as well as a content control: a crawler that ignores robots.txt and gets denied at the edge costs nothing, while the same crawler rendering pages through your functions costs real compute. If you are still choosing where to host, our ranking of where to deploy your app in 2026 covers which platforms give you this kind of edge control on their cheaper tiers.

Cloudflare's equivalent is more granular and, since September 15, 2026, organized by the same three jobs this guide uses. Its AI Crawl Control lets you set Search, Agent and Training separately, each to allow, block everywhere, or block only on pages that display ads, and a new Disallow AI Training setting publishes a no-training preference in robots.txt while keeping "accountable" mixed-use crawlers like Googlebot, Bingbot and Applebot allowed for search - Cloudflare. Training-only crawlers from Amazon, Anthropic, Meta and OpenAI are blocked at the network under that setting. The onboarding screen below shows the defaults Cloudflare now recommends to a site that monetizes with ads.

The screenshot captures the whole philosophy of this guide in one form: Search stays on, Training is disallowed, and the Agent setting blocks only on pages that carry ads, where an agent reading the page on someone's behalf would strip out the ad revenue. The toggle at the bottom is Bot Preference Sync, which writes your dashboard choices into robots.txt so the published file matches the enforced policy, and which is on by default for new customers - Cloudflare. The managed robots.txt it writes also carries Cloudflare's Content-Signal: search=yes, ai-train=no line - Cloudflare Docs.

If you run your own servers, the same idea is a few lines of web server configuration. The block below denies the declared training crawlers by user agent in nginx and leaves everything else untouched; the ai.robots.txt project publishes longer, maintained versions for nginx, Apache, Caddy, HAProxy and lighttpd.

map $http_user_agent $ai_training_bot {
    default 0;
    ~*(GPTBot|ClaudeBot|meta-externalagent|Amazonbot|CCBot|Bytespider) 1;
}

server {
    location = /robots.txt { }          # always let crawlers read your preferences
    location / {
        if ($ai_training_bot) { return 403; }
        # ... your normal configuration
    }
}

Note the exemption for /robots.txt. A crawler that cannot read your file cannot learn your preference, which is exactly the failure Anthropic warns about when sites block its IP addresses instead of naming its token. Note also what is missing from the map: Google-Extended and Applebot-Extended are tokens, not crawlers, so there is no user agent to deny. Blocking Googlebot or Applebot at the firewall would remove you from their search products, so for Google and Apple, robots.txt is the only training control available.

7.3 Verify before you trust a user agent

A user agent is a string anyone can type. If you build allow rules (for example, letting OAI-SearchBot through a strict firewall), verify the request really comes from the operator. The major labs publish their crawler IP ranges as JSON: OpenAI lists separate files for OAI-SearchBot, GPTBot and ChatGPT-User - OpenAI. Perplexity does the same for its two agents - Perplexity. Matching the claimed user agent against the published range stops the cheapest kind of impersonation.

The more durable answer is cryptographic. Web Bot Auth lets a crawler sign its requests with HTTP message signatures so a site can verify the operator without trusting IP lists. Google added documentation in May 2026 for verifying requests signed under its agent.bot.goog identity, while noting that the feature is experimental and that not every request is signed yet - Google. We covered why agents need verifiable identities rather than shared keys in our guide to giving your AI agent an identity; the same logic now applies to the crawlers visiting your site.

7.4 What enforcement cannot do

Enforcement has limits worth stating plainly. It cannot reach content that was fetched before you deployed it, it cannot stop a model from learning about you from other sites that quote you, and it cannot distinguish a disguised scraper from a person with perfect accuracy. Aggressive bot challenges also have a cost: every real visitor who meets a challenge page is a little friction, and some automated visitors (an agent booking a demo for its user) are customers. Our pre-launch security checklist covers the abuse-protection side of this balance for new products.

So the sensible enforcement posture is proportional. Deny declared training crawlers at the edge if training is genuinely against your interest. Rate-limit rather than block when the problem is load. Reserve behavioral challenges for paths where scraping actually hurts (pricing data, proprietary listings, account pages) rather than putting your whole site behind them. And keep the robots.txt file reachable to everyone, because it is the one place every compliant operator looks for your intent.

8. Google Is the Hard Case: Training, AI Overviews and Search

Google is where "block training, keep AI search" gets complicated, because Google's AI search is not a separate crawler. AI Overviews and AI Mode are built from the same Search index that Googlebot crawls, so there is no robots.txt token that removes you from AI Overviews while keeping you in classic results. Google-Extended does not help either: it governs Gemini model training and grounding in the Gemini app and Vertex AI, and Google says it does not affect Search inclusion or ranking - Google. Until mid-2026, the only way out of AI Overviews was nosnippet or leaving Search altogether.

That changed in 2026, under regulatory pressure. Google now offers three separate controls, and you need to understand all three to set the policy you actually want. They answer three different questions: may Google train on my content, may Google's AI answers in Search use and link my content, and may my pages appear in Search at all.

8.1 The three Google controls, side by side

The first control is Google-Extended in robots.txt, covering training of future Gemini models and grounding in Gemini Apps and Vertex AI. The second is the Search generative AI control in Search Console, which Google rolled out to all websites worldwide as of August 31, 2026: setting it to exclude removes your site's links and content from AI Overviews, AI Mode and generative AI features in Discover, it "isn't used as a ranking or inclusion signal" for the rest of Search, and it "doesn't affect AI training" - Google Search Console Help. The third is ordinary noindex, which removes a page from Search entirely.

The table makes the trade-off concrete. If your goal is "no training, but keep every kind of Google traffic," you disallow Google-Extended (accepting the loss of Gemini app grounding) and leave the Search Console control at Include. If your goal is "keep training and Search, but no AI Overviews," you do the reverse. Google's own numbers explain why few sites choose the second: AI Overviews has more than 2.5 billion monthly active users and AI Mode more than one billion, and sites that opt out "will not receive traffic or impressions" from those features - Google.

8.2 Measure before you decide

The same rollout added the measurement that makes this decision rational instead of emotional. Search Console now has Generative AI performance reports for Search and Discover, showing impressions in AI Overviews, AI Mode and Discover's AI features, broken down by page, country, device and date - Google Search Central. Clicks are not included yet, but impressions by page already tell you which of your pages Google's AI is using, which is exactly the evidence you need before excluding them.

The report in the screenshot is the place to start every AI Overviews conversation. Look at which pages carry the impressions: if they are pages whose whole value is the answer (a definition, a price, a statistic), AI Overviews may be substituting for your visit, and the exclude control is worth testing. If they are product, category or brand pages, the impression is advertising you did not pay for. For a deeper treatment of getting cited and placed inside Google's AI surfaces, see our guide to Google's Connected Apps Platform.

8.3 Why Google moved: the CMA and the "accountable" commitments

The Search Console control exists because the UK's Competition and Markets Authority required it. On June 3, 2026, under the Digital Markets, Competition and Consumers Act, the CMA imposed a conduct requirement on Google Search obliging it to let websites opt out of AI search features such as AI Overviews and AI Mode, to let them opt out of training, and to attribute publisher content with clear links, with nine months to introduce page-level controls - Search Engine Journal. Google tested the toggle with UK site owners first and then rolled it out globally.

Cloudflare's "accountable mixed-use crawler" designation adds a second set of commitments. To qualify, an operator must offer a robots.txt training opt-out, an AI summary opt-out, URL-level visibility into which pages were used for training, and assurance that a training opt-out does not affect search; Cloudflare names Googlebot, Bingbot and Applebot as accountable, and says Google is building URL-level transparency tools for Google-Extended - Cloudflare. The direction is clear: by 2027, you should be able to see, page by page, which of your pages each mixed-use operator used for what.

8.4 Bing's version: grounding is its own decision

Microsoft splits the same problem differently. Training will follow the shared token and a page-level tag, while grounding (using your page to build an answer in Copilot or Bing's AI experiences) has its own page-level control:

<!-- Opt this page out of Microsoft AI training (read from Feb 1, 2027) -->
<meta name="Bingbot" content="ai-training:disallow">

<!-- Keep this page out of AI answer grounding, or limit it to Microsoft's own products -->
<meta name="Bingbot" content="ai-grounding:disallow">
<meta name="Bingbot" content="ai-grounding:disallowextended">

Both preferences default to allow when nothing is published, and the same values can be sent as X-Robots-Tag: Bingbot: ai-training:disallow headers for non-HTML files - Microsoft Bing. If you currently use noarchive to keep pages out of Bing's AI answers, March 1, 2027 is a real deadline: after it, noarchive no longer counts, and pages without the new grounding tag (or a temporary preference recorded in Bing Webmaster Tools) default to allow until recrawled. For "block training, keep AI search," the Bing setup is the shared token in robots.txt and no grounding tag at all.

9. What Founden Learned Blocking Training Crawlers, Then Welcoming Them

Founden builds autonomous software companies: each one is a running product with its own site, back office and domain, built by an AI builder that can keep developing it, and Founden hosts their sites. That gives Founden an unusual vantage point on this question, because a robots.txt policy chosen once at the platform level ships to every company it builds. This section reports what that policy was, why it changed, and what the measurements showed, because the reversal is the most direct evidence we have on the question in this article's title.

From June 13, 2026, every new company Founden built shipped a robots.txt that did exactly what this guide's title describes. Its own source comment stated the policy: allow traditional and AI search crawlers "(they power AI answers that cite this site)", and block AI training crawlers "(keep the company's content out of model training sets)". The block named eleven tokens (GPTBot, Google-Extended, CCBot, ClaudeBot, anthropic-ai, Bytespider, Diffbot, Applebot-Extended, Omgilibot, ImagesiftBot and Meta-ExternalAgent), while OAI-SearchBot, ChatGPT-User, PerplexityBot, Googlebot, Bingbot and Applebot fell through to the default allow.

9.1 Why the default flipped on November 1

On November 1, 2026, the platform replaced that policy with a different rule, decided by Founden's founder: a crawler whose owner sends these sites nobody is refused, and every other crawler is welcome, including the training crawlers of the labs whose assistants send people. The reasoning recorded in the new robots.txt source is a first-principles one about how assistants answer: an assistant names the brands its model learned, and searches for the ones it already knows, so a site no model trained on is rarely named in an answer, from memory or with search. The live page, not a frozen copy in a training set, remains the source an assistant cites.

That rule fits the companies Founden builds. They are young products whose problem is being unknown, not being copied. For them, a model that has learned their name, category and audience is distribution, and the training crawl that teaches it is the cheapest marketing they will ever get. The rule also kept something the old policy missed entirely: the crawlers that were actually costing money were mostly not AI labs at all.

9.2 What the measurements showed

Before the new rule went live, Founden measured who was requesting pages across its 59 busiest sites, about 3.43 million requests a day, and cross-checked every company's referrers over 30 days (290,921 page views across the fleet). The table summarizes the request mix by who was asking.

The refused group is where the "sends nobody" test bites. Reflectionbot, Reflection AI's model-training crawler, made about 30,960 requests a day across 21 sites and sent zero visits in 30 days. Reflection AI is a signatory of the EU's AI Code of Practice, which commits it to honoring robots.txt, so a robots.txt disallow is enough to stop it - European Commission. AhrefsBot made about 41,220 requests a day across 15 sites and its 44 attributed visits came from people using the SEO tool. ElevenlabsCrawler (about 14,640 a day, gathering knowledge for its customers' voice agents) and Huawei's PetalBot (about 10,140 a day on 18 sites) sent none.

The welcomed group included GPTBot, ClaudeBot, CCBot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta's agents and the search engines. They used more requests and more function time than the refused crawlers, and that was a deliberate choice: their owners run the assistants and search engines through which these companies get found. The largest automated cost on the fleet was not a named crawler at all but programs driving old versions of desktop Chrome, which the platform now challenges separately, and which no robots.txt line could ever have touched.

9.3 How the refusal is enforced, and how one site went further

The refused list lives in one file that two things read. The host firewall builds a single deny rule from it, applied on every path except /robots.txt, so a refused crawler is turned away before the site runs (and, on Vercel, a denied request is not billed). Each company's robots.txt tells the same names Disallow: /, so a crawler that reads the file stops asking. A company can lift a default for its own site, for example if its owner uses an SEO tool on it, and since November 3 it can also refuse a crawler the platform welcomes.

The first company to use that refusal was AITraining.jobs, a job board for paid AI training work that Founden's founder runs on the platform. Its own measurement found that ExaSearchBot, a search crawler the platform welcomes because its customers' AI answers can send readers, took 8.6% of the site's function time, ignored the 60-second crawl delay its robots.txt asked for, and referred no visit in 30 days. On its founder's decision, ExaSearchBot and ExaUser are now refused on that one site, while every other company keeps welcoming them. That is the rule working as intended: welcome by default, refuse on evidence, and let the evidence be per site.

9.4 What this case says about your own policy

The Founden experience does not prove that blocking training is wrong. It shows that the question is the wrong shape. "Block AI training, keep AI search" treats the labs as the problem, when for a young company the labs' crawlers are among the few automated visitors with a path back to a human reader. The crawlers that were pure cost (SEO tools, data resellers, training crawlers for products nobody reaches the site through) were never in the original block list at all, because that list was built by category instead of by return.

The practical version for your own site is to run the same test on your own logs. List the top automated user agents by requests, find each owner, and ask whether that owner has any way to send you a person: a search engine, an assistant with citations, a link preview, a directory. Refuse the ones that cannot. Then make the training decision separately, based on whether your content is your product or your advertisement, which is the subject of the next section.

10. A Decision Framework: Should You Block AI Training at All?

Start from what your content does for your business, not from how you feel about AI companies. Every page on the web does one of two economic jobs. Either the content is the product (people pay, subscribe or watch ads to read it, and an answer that reproduces it substitutes for the visit), or the content is the advertisement for a product sold somewhere else (a pricing page, documentation, a product listing, a founder's blog), and an answer that repeats it creates demand. Most sites are a mix, which is why path-level rules exist.

Training is the use that most directly substitutes for content-as-product, because a model that has learned your analysis can paraphrase it forever without retrieving your page or citing you. For content-as-advertisement, training is the opposite: free, durable placement in the memory of the systems people now ask for recommendations. The question "should I block AI training?" therefore has a different correct answer for a news site and for a software company, and the same answer can differ between /research/ and /pricing/ on the same domain.

The diagram's lower half is the same for every site, and it is the part people skip. Whatever you decide about training, keep search crawlers (they are how AI answers cite and link you), refuse crawlers that can never send you a reader, and add edge enforcement in proportion to how valuable your content is to someone who ignores the rules. Only the top half, the training decision, depends on your business model.

10.1 Seven sites, seven decisions

A news or analysis publisher whose revenue depends on people reading the article should opt out of training with every token in Template A, keep every search crawler, consider Bing's grounding tag and Google's Search Console exclusion for the pages where AI answers clearly substitute for visits, and pursue licensing (Section 11) for the rest. This is the case the shared token was designed for, and the Zhao and Berman evidence suggests these sites should still be careful not to over-block.

A software company is the opposite case. Its docs, comparisons and pricing pages exist to be read and repeated; an assistant that knows its name and what it does is a sales channel. It should welcome training crawlers from labs whose assistants send people, keep search crawlers, and invest in being cited, which we covered in depth in our guide to getting your site cited by ChatGPT and Claude. An online store sits close to the software case: product pages are advertisements, and being named in shopping answers is the goal, as our guide to getting products recommended by ChatGPT explains. The store's training decision matters less than its search and product-feed decisions.

An independent creator (a photographer, illustrator or writer selling their own work) has content-as-product in its purest form and should opt out of training everywhere, including image-specific crawlers, and enforce at the edge if their work is scraped heavily. A data or directory business (job listings, prices, reviews aggregated from many sources) has a subtler choice: the dataset is the product, but being the site assistants name for that category is the moat. AITraining.jobs, from the case above, chose to welcome lab training crawlers while refusing a crawler that took without returning. A course creator splits by path: the marketing pages welcome training, the lesson content opts out.

A local service business (a clinic, a law office, a contractor) is the easiest case of all. Its pages describe services, prices and locations that it wants every assistant to repeat accurately, and nothing on them is worth paying to read. It should welcome training and search crawlers alike, and spend its effort making sure the facts a model learns (hours, service area, prices) are current on the site, because a model that learned an old price will repeat it until the next training run.

The pattern underneath these cases is that the training decision follows the money, while the search decision is almost always "keep." The few exceptions, sites that genuinely want out of AI answers too, are publishers whose pages are answers, and they now have Google's Search Console control and Bing's grounding tag for exactly that purpose. If you build software with AI tools, note that a separate training question applies to your code and prompts inside those tools; our guide to AI builder training opt-outs covers that side.

11. The Legal Layer: What an Opt-Out Is Worth in 2026

robots.txt began as a courtesy between webmasters and crawler operators, and in most of the world it is still not law. RFC 9309 itself states that its rules "are not a form of access authorization" - RFC Editor, and Google's own robots.txt video describes compliance as a matter of crawlers being well behaved. What has changed is that, in Europe especially, a machine-readable opt-out has become evidence with legal weight, and the shared token is likely to carry more of that weight than the proprietary names it complements.

The mechanism is the EU's text and data mining rules, under which rightsholders can reserve their content from mining in a machine-readable way, and the General-Purpose AI Code of Practice built on top of them. The Code recognizes robots.txt as a way to reserve those rights and commits signatories to identifying and complying with other appropriate machine-readable protocols, including those adopted by international or European standardisation organisations - Paul, Weiss. The signatory list includes Amazon, Anthropic, Google, IBM, Microsoft, Mistral AI and OpenAI; Meta is not on it, and xAI signed only the safety and security chapter - European Commission.

11.1 Why an IETF token matters more than a vendor token

Put those two facts together and the strategic value of __usage_ai-training becomes clear. A proprietary token like GPTBot is honored because OpenAI says so. A token that comes out of an IETF process is the kind of "appropriate machine-readable protocol" the Code's signatories commit to honoring, which turns a voluntary courtesy into part of a documented compliance commitment. Once the AI Preferences vocabulary and the training token mature, an EU-facing site that publishes the shared token has a stronger record that it reserved its rights than one that relied on a list of company names.

European courts are also narrowing what training can do even with lawfully accessed content. In GEMA v OpenAI, the Munich Regional Court held on November 11, 2025 (case 42 O 14139/24) that the text and data mining exception covers training only so long as the model does not memorize protected works, and that memorized song lyrics reproduced in outputs infringed copyright; OpenAI has said it will appeal - Taylor Wessing. If you build an AI product yourself, the compliance side of these rules applies to you too, which we cover in our guide to making an AI app EU-compliant.

11.2 Licensing: when the answer is "pay me" rather than "no"

For sites whose content is valuable enough that AI companies would pay for it, blocking is a negotiating position rather than an end state. Two standards now let you publish terms in a machine-readable way. Really Simple Licensing (RSL) reached version 1.0 on December 10, 2025, with an XML vocabulary for usage, licensing and legal terms that complements robots.txt, backed by Cloudflare, Akamai, the Associated Press and Stack Overflow among others - The Register. Pointing crawlers to your terms is one line, and each subdomain needs its own - RSL:

License: https://example.com/license.xml

Cloudflare's pay per crawl goes further and attaches a price to each request: a crawler either signals payment intent and receives the page, or receives an HTTP 402 Payment Required response with the price, and publishers can allow, charge or block each crawler individually - Cloudflare. It verifies paying crawlers with Web Bot Auth signatures so the payment cannot be spoofed. Both approaches share a limitation that the scoring table reflects: they only produce money when AI companies agree to pay, and outside of large publishers, few have.

The Kelley draft anticipates this world explicitly: it says private agreements may override preferences expressed through the shared token - IETF. That is the right mental model for a publisher. The public robots.txt says "no training" to everyone; a signed license says "yes, on these terms" to the companies that pay. Without the public opt-out, there is nothing to license.

12. Monitoring: Prove Your Policy Works

A robots.txt policy you never check is a guess. The operators change their crawlers, rename user agents and add new ones several times a year: Google alone changed its NotebookLM user agent to Google-GeminiNotebook in July 2026 and added the Google-Agent fetcher in March - Google. The question monitoring answers is simple: are the crawlers you blocked still requesting pages, are the crawlers you kept still visiting, and are people arriving from the assistants you care about?

Three data sources answer it, and you probably already have all of them. Your server or CDN logs show requests by user agent. Your analytics show visits by referrer, which is where AI assistant traffic appears. And Google's Search Console now shows AI Overviews and AI Mode impressions by page. None of them requires a vendor, and all three together tell you whether your policy is doing what you meant.

12.1 Read your logs by user agent

Start with a count of requests by crawler over the last week. On a server with standard access logs, one command gives you the picture; on Vercel or Cloudflare, the same breakdown is available in the firewall and analytics dashboards.

# Requests per AI crawler in an nginx/Apache combined log
grep -oiE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|meta-externalagent|meta-webindexer|Amazonbot|Amzn-SearchBot|CCBot|Bytespider|Googlebot|bingbot|Applebot" access.log \
  | sort | uniq -c | sort -rn

# Did a blocked crawler still get pages? (look for 200s after your change date)
grep -i "GPTBot" access.log | awk '$9 == 200 {print $4, $7}' | tail -20

Interpret the results with the caches in mind. A blocked crawler that keeps fetching for a day after your change is reading a cached copy of your old file; one that keeps fetching a week later is either ignoring the rule or matching a group you did not intend. A search crawler whose requests drop sharply after a change is the red flag that matters most, because it usually means a rule caught more than you meant. If the counts look wrong, test the file again with the parser from Section 6.5 before blaming the crawler.

12.2 Read your referrers and Search Console

On the traffic side, segment your analytics by referrer domain for the assistants: chatgpt.com, perplexity.ai, gemini.google.com, claude.ai and copilot.microsoft.com cover most of what Statcounter measures. Track them as one channel with a weekly total, and annotate your robots.txt changes on the same chart. Because training's return arrives as direct and branded search traffic rather than as an AI referral, watch branded search impressions in Search Console alongside it; a training opt-out that works perfectly may still show up months later as fewer people searching for your name.

Then open the Generative AI performance report from Section 8.2 and sort pages by impressions. Those are the pages Google's AI is using, and the list is the factual basis for every AI Overviews decision. For teams that want this automated, several of the tools in our technical SEO agent skills roundup can run log, robots.txt and sitemap checks on a schedule. Revisit the policy itself whenever a major operator updates its crawler documentation, and at least once a quarter; in 2026, that cadence would have caught every change in this guide.

13. Future Outlook: Grounding, Signed Agents and Paid Crawling

The October 30 commitments settle the training question's syntax, but training is the least of where AI crawling is going. The next frontier is grounding: whether an assistant may read your page at the moment it answers a question, quote it, and summarize it in place of a visit. Microsoft already separates grounding from training at the page level, Google's Search Console control covers its AI answers, and Cloudflare's accountable designation requires an AI summary opt-out that it says site owners will be able to set through Cloudflare next year - Cloudflare. Expect the IETF vocabulary to carry that category into robots.txt the same way it carried training, and expect the debate there to be harder, because grounding is also how citations and links are made.

The second shift is that more of your automated visitors will be agents acting for specific people rather than crawlers acting for a company. Cloudflare reported in July 2026 that agent traffic now makes up more than half of internet traffic - Cloudflare, and Google, OpenAI and others are moving those agents onto signed identities with Web Bot Auth. That is good news for site owners: a signed agent can be verified, rate-limited and, if you choose, charged, without guessing from user-agent strings. It also means robots.txt will matter less for this traffic over time, because a person's agent is a visitor, and the controls that fit visitors are product decisions (what an agent may do on your site, and what it costs) rather than crawl directives.

13.1 From blocking to distribution

The third shift is the one most sites are underweighting. As discovery moves into assistants, the question changes from "what may AI companies take from my site" to "how does my product show up inside the places people now ask." Google's Connected Apps Platform, MCP servers that let assistants use your product directly, and agent skills that teach agents how to work with it are all ways of being present by design rather than by crawl. We have covered each: shipping an MCP server, shipping an agent skill and selling inside ChatGPT. A robots.txt decision is defensive; these are the offensive half of the same strategy.

Paid crawling will grow too, but slowly and unevenly. Pay per crawl and RSL give publishers a price tag and a machine-readable license; the open question is how many AI companies will pay at prices that matter, and so far the answer is mostly the largest publishers through direct deals. For most sites, the realistic 2027 setup is the one this guide builds: a shared training token plus per-company tokens in robots.txt, search crawlers kept, crawlers that send nothing refused, edge enforcement in proportion to the content's value, and a quarterly look at the logs to keep it all true.

Conclusion: Separate the Decisions, Then Measure

The October 30 announcements make the mechanics of a training opt-out easier than they have ever been: two lines that every major lab has said it will read, alongside the per-company tokens that work today. What they do not do is make the decision for you. The single most important idea in this guide is that "AI crawlers" are three different things (training, search and user-triggered fetching), and each deserves its own answer.

For most sites, the search answer is "keep," because search crawlers are how AI answers cite and link you, and blocking them costs measurable traffic. The user-triggered answer is usually "treat it like a visitor," because there is a person behind it. The training answer depends on what your content does for your business: opt out if the content is your product, welcome training if the content advertises a product you sell, and decide per path if you have both. Then refuse the crawlers whose owners can never send you anyone, and back the whole policy with edge enforcement if your content is worth ignoring the rules for.

Founden's own reversal is a useful check on any of these choices. Its sites shipped "block training, keep search" for four and a half months, and the measurements showed the real cost sat with crawlers that were never on the training block list, while most of the training crawlers it had blocked fed the very assistants that send its companies readers. Whatever you choose, choose it from your own logs and referrers, not from a list someone else wrote, and revisit it when the operators change their documentation. In 2026, that happened often.

This guide reflects crawler documentation, standards drafts and data as of November 4, 2026. The shared training token is an Internet-Draft, not a finished standard, and no provider has confirmed it is live yet; crawler names, provider policies and the figures quoted here change frequently, so verify current details with each operator's documentation before relying on them.

In this topic: Getting customers

• Google Connected Apps Platform: Gemini + Search (2026)
• How to Create Agent Skills for Your Product (2026)
• Get Your Products Recommended by ChatGPT 2026
• Sell Your Product Inside ChatGPT: 2026 Guide
• Generate a Month of Ad Creative for $5 (2026)
• Sell to AI Agents: 2026 Setup Guide
• Get Your Site Cited by ChatGPT and Claude 2026
• How to Start a Community: 2026 Founder's Guide

How do you get found and get customers? Every guide on it

Data behind this topic: Startup and AI directories: what each did with our listings.