Share of answer is the operating metric for this practice.

How we count it
Request an audit

Insight · 13 min

robots.txt for AI crawlers

AI crawlers come in three kinds: training crawlers that collect pages for future models, search crawlers that index pages so an assistant can cite them, and user-triggered fetchers that load a page because a person asked. Each operator gives each kind its own robots.txt token, so blocking GPTBot does not block OAI-SearchBot or ChatGPT-User. Decide per kind, write one group per token, and check the result in your server logs.

In short

  • Training, search, and user-triggered bots are separate tokens. A rule for one does not reach the others.
  • Blocking a search token (OAI-SearchBot, Claude-SearchBot, PerplexityBot) removes you from that assistant’s cited answers. Blocking a training token does not.
  • Google-Extended and Applebot-Extended are control tokens, not crawlers. Google-Extended does not control AI Overviews. Googlebot does.
  • Several user-triggered fetchers say they may ignore robots.txt. robots.txt is a request under RFC 9309, not access control.
  • CDN settings can block bots before robots.txt is read. Check them, because they can override the policy you wrote.

What are the three kinds of AI crawler?

The three kinds are training crawlers, search crawlers, and user-triggered fetchers, and they do different things with your page.

  1. Training crawlers collect pages that may be used to train future models. GPTBot, ClaudeBot, and MistralAI-Training are examples. Blocking them affects what future models learn. It does not affect whether today’s assistant can cite you.
  2. Search crawlers build the index an assistant searches before it answers. OAI-SearchBot, Claude-SearchBot, and PerplexityBot are examples. Blocking them is the closest thing to opting out of AI citations on that platform.
  3. User-triggered fetchers load one URL because a user pasted it, asked about it, or sent an agent to it. ChatGPT-User, Claude-User, and Perplexity-User are examples. They are not crawlers in the usual sense, and several operators say robots.txt may not apply to them.

The tokens are independent by design. OpenAI states that each setting is independent, and that a site can allow OAI-SearchBot while disallowing GPTBot. The same split exists at Anthropic, Perplexity, Meta, Amazon, and Mistral. A single Disallow aimed at “AI” does not exist. You write a rule per token or you write User-agent: * and block everyone, including Googlebot.

Which AI bots exist, and which ones honor robots.txt?

The table below lists the tokens each operator documents as of November 2026, with the operator’s own statement on robots.txt. “Honors” means the operator says so in its documentation. It is not an independent audit.

TokenOperatorKindWhat the operator says it doesrobots.txt, per the operatorDoc
GPTBotOpenAITrainingCrawls content that may be used to train generative AI foundation modelsHonors. Disallowing it signals content should not be used for trainingOpenAI
OAI-SearchBotOpenAISearchSurfaces websites in ChatGPT search featuresHonors. Opted-out sites are not shown in ChatGPT search answers but can still appear as navigational links. About 24 hours to take effectOpenAI
ChatGPT-UserOpenAIUser-triggeredUser actions in ChatGPT and Custom GPTs, including GPT Actions“robots.txt rules may not apply.” Not used to decide search inclusionOpenAI
OAI-AdsBotOpenAIOtherVisits only pages submitted as ads on ChatGPT, to check them. Not used for trainingNot statedOpenAI
ClaudeBotAnthropicTrainingCollects content that may be used to train Anthropic’s modelsHonors, including the non-standard Crawl-delayAnthropic
Claude-SearchBotAnthropicSearchIndexes the web to improve search result quality for Claude usersHonorsAnthropic
Claude-UserAnthropicUser-triggeredFetches pages when a Claude user asksHonors. Anthropic states its bots honor robots.txt and that blocking Claude-User prevents retrieval for user queriesAnthropic
PerplexityBotPerplexitySearchSurfaces and links websites in Perplexity search results. Not used to crawl for foundation modelsManaged through robots.txt. Up to 24 hours to take effectPerplexity
Perplexity-UserPerplexityUser-triggeredSupports user actions in Perplexity. Not used for crawling or training“generally ignores robots.txt rules”Perplexity
GooglebotGoogleSearchCrawls for Google Search, including AI Overviews and AI ModeHonors. Google calls Googlebot rules “the control” for AI features in SearchGoogle
Google-ExtendedGoogleControl tokenGoverns whether crawled content is used to train future Gemini models and for grounding in Gemini Apps and Vertex AIToken only. No separate user agent. No effect on Search inclusion or rankingGoogle
Google-AgentGoogleUser-triggeredAgents on Google infrastructure that browse and act on a user’s requestUser-triggered fetchers “generally ignore robots.txt rules”Google
Google-GeminiNotebookGoogleUser-triggeredFetches URLs users add as notebook sources. Replaces Google-NotebookLMGenerally ignores robots.txtGoogle
ApplebotAppleSearch, also trainingPowers Spotlight, Siri, and Safari search. Its data “may also be used to help train Apple foundation models”Honors. Ignores Crawl-delay. Falls back to Googlebot rules if none name ApplebotApple
Applebot-ExtendedAppleControl tokenDecides whether Applebot’s data trains Apple’s generative modelsToken only. “Applebot-Extended does not crawl webpages.” Not used in Search rankingApple
CCBotCommon CrawlOpen web archiveBuilds the public Common Crawl datasetOpt out with a CCBot Disallow: / group. Warns that other crawlers impersonate CCBotCommon Crawl
Meta-ExternalAgentMetaTraining, also indexingCrawls to train foundation AI models or to improve products by indexing contentMeta documents robots.txt rules to block itMeta
Meta-WebIndexerMetaSearchImproves Meta AI search results. Allowing it helps Meta “cite and link to your content”Meta documents robots.txt managementMeta
Meta-ExternalFetcherMetaUser-triggeredFetches links at a user’s request for agentic AI features“may bypass robots.txt rules”Meta
AmazonbotAmazonMixedImproves Amazon products and services. “may be used to train Amazon AI models”Honors. Reads a noarchive robots meta tag as “do not use the page for model training.” No Crawl-delayAmazon
Amzn-SearchBotAmazonSearchSearch experiences in Amazon products, including Alexa. Not used for trainingHonors. Follows rules for other search bots if none name itAmazon
Amzn-UserAmazonUser-triggeredLive fetches to answer customer questions. Not used for training“may not follow all robots.txt directives”Amazon
DuckAssistBotDuckDuckGoSearchFetches pages in real time for DuckDuckGo’s AI-assisted answers. “not used in any way to train AI models”Honors. 72 hours to take effect. No effect on organic rankingsDuckDuckGo
MistralAI-TrainingMistralTrainingBuilds datasets to train Mistral modelsCan be disallowed in robots.txtMistral
MistralAI-IndexMistralSearchIndexes pages for Mistral search. Not used for trainingNot describedMistral
MistralAI-UserMistralUser-triggeredFetches a page when a user asks Mistral’s assistant, and links the sourceThe token “governs which sites these user requests can be made to”Mistral

Three notes on the table.

Google-Extended does not cover AI Overviews or AI Mode. Google states that AI features are part of Search, that Googlebot rules are the control, and that nosnippet, data-nosnippet, max-snippet, and noindex limit what Search shows. Google-Extended limits training and grounding in Gemini Apps and Vertex AI. To stay out of AI Overviews, you limit snippets or indexing in Search, and that costs you in classic results too.

Applebot is a mixed crawler. Apple says Applebot data may help train its foundation models, and that disallowing Applebot-Extended is the opt-out. Blocking Applebot itself removes you from Spotlight, Siri, and Safari search.

Operator statements are claims. In August 2025, Cloudflare reported that Perplexity, when its declared crawler was blocked, fetched pages with an undeclared user agent impersonating Chrome on macOS, from IPs outside its published ranges. Cloudflare removed Perplexity from its verified bot list. If a bot’s behavior matters to you, enforce it at the network edge, not only in robots.txt.

Does robots.txt actually stop a crawler?

No. robots.txt is a request a crawler chooses to honor. RFC 9309, the Robots Exclusion Protocol standard published in September 2022, says its rules “are not a form of access authorization” and are “not a substitute for valid content security measures.”

The RFC also sets behavior worth knowing:

  • User-agent matching is case-insensitive. A product token may contain only letters, underscores, and hyphens, so User-agent: GPT* is not a valid way to match several bots.
  • When several rules match a path, the most specific one (the longest) wins.
  • Crawlers should not use a cached robots.txt for more than 24 hours, unless it is unreachable.
  • A 4xx response for robots.txt means the crawler may fetch anything. A 5xx response means it must assume everything is disallowed.

That last pair matters. A robots.txt that returns 404 allows everything. A robots.txt that returns 500 during an outage blocks everything, for every compliant bot, until it recovers.

Anything that must stay private needs authentication. A paywall, a login, or an IP allowlist is access control. A Disallow line is a note on the door.

What do CDN AI-crawler settings do?

CDN settings block or allow bots at the network edge, before the request reaches your server and whether or not the bot reads robots.txt. Cloudflare is the most common case.

As of November 2026, Cloudflare’s “AI Crawler and Agent access policies” sort bots into three categories, Search, Training, and Agent, each with its own action. Cloudflare’s developer documentation, last updated November 2, 2026, lists Allow as the default for new domains. Cloudflare’s September 15, 2026 announcement says new domains onboarding from that date are offered presets: ad-supported sites get Training set to “Disallow AI Training” and Agent set to block on pages with ads, while other sites get Allow. Existing settings carried over.

Two details change outcomes:

  • “Block” and “Block on pages with ads” now apply to mixed-use crawlers, including Googlebot, Applebot, and Bingbot. Choosing Block for Training can therefore block Googlebot from Search.
  • “Disallow AI Training” publishes a robots.txt directive instead of blocking at the edge. Mixed-use crawlers keep crawling for search under that setting.

Cloudflare announced one set of new defaults in July 2026 and shipped a different set in September. Open the dashboard and read the current values. A robots.txt that allows OAI-SearchBot does nothing if the edge returns 403 to it.

What should my robots.txt say?

Your robots.txt should say what you decided for each kind of crawler. Three common policies follow. Each one costs something.

Policy 1: allow all

# Allow every crawler, including AI training, search, and user fetchers.
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Cost: none in visibility. Your public pages can enter training sets for future models from OpenAI, Anthropic, Google, Apple, Meta, Amazon, Mistral, and Common Crawl.

Policy 2: allow search and user fetches, block training

# Training crawlers and control tokens: opt out.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: MistralAI-Training
Disallow: /

# Everyone else, including OAI-SearchBot, Claude-SearchBot,
# PerplexityBot, Googlebot, Applebot, and user-triggered fetchers.
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Cost: small, and real. Search and citation stay open. Future models learn less about you from your own pages, so their unprompted knowledge of your brand depends more on third parties. Blocking Google-Extended also opts you out of grounding in Gemini Apps and Vertex AI, which is retrieval, not only training. Meta-ExternalAgent is documented as both training and indexing, so blocking it may also reduce Meta AI coverage. Amazonbot is mixed too: leave it allowed and add a noarchive robots meta tag if you want Amazon to skip training on a page.

Policy 3: block all AI

# Training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: MistralAI-Training
# AI search
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Meta-WebIndexer
User-agent: Amzn-SearchBot
User-agent: DuckAssistBot
User-agent: MistralAI-Index
# User-triggered
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Meta-ExternalFetcher
User-agent: Amzn-User
User-agent: MistralAI-User
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Cost: large. You disappear from cited answers in ChatGPT search, Claude, Perplexity, Meta AI, Alexa, DuckDuckGo, and Mistral. Assistants will still describe you, from training data and from other sites, and you will have no say in which. The policy does not remove you from Google AI Overviews, because Googlebot is still allowed. Several user-triggered fetchers say they may ignore these rules. If blocking is a hard requirement, back it with edge rules.

Policy 2 is a reasonable default for most companies that sell something. Policy 3 is a business decision, usually a publisher’s, and it belongs in writing with an owner. See what gets cited for why fetchable pages matter to generative engine optimization.

How do I check what the bots actually do?

Server logs show which bots arrived, what they requested, and what status they got. robots.txt shows only what you asked for.

  1. Count requests by token. On a combined-format access log:
grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Google-Agent|Applebot|CCBot|meta-externalagent|meta-webindexer|meta-externalfetcher|Amazonbot|Amzn-SearchBot|Amzn-User|DuckAssistBot|MistralAI-[A-Za-z]+" access.log \
  | sort | uniq -c | sort -rn
  1. Check status codes per token. A search bot you allowed should get 200s, not 403s from a CDN or WAF:
grep "OAI-SearchBot" access.log | awk '{print $9}' | sort | uniq -c
  1. Confirm robots.txt fetches. Each compliant bot requests /robots.txt. OpenAI says it may add a robots.txt marker to its user-agent string on those requests.
  2. Verify the IP, not the string. User-agent strings are easy to fake, and Common Crawl and Google both warn about impersonation. The next section shows how.
  3. Allow for lag. OpenAI and Perplexity cite about 24 hours for a robots.txt change to take effect. DuckDuckGo cites 72 hours. RFC 9309 caps the cache at 24 hours for compliant crawlers.

How do I verify that a request really came from the bot it names?

Check the source IP against the operator’s own record. Google documents a DNS check. OpenAI, Anthropic, and Perplexity publish IP lists instead. None of the three documents reverse DNS as a verification method, so for them the list is the test.

OperatorDocumented methodWhere
GoogleReverse DNS on the IP, then a forward lookup on the hostname. The hostname must end in googlebot.com, google.com, or googleusercontent.com, and the forward lookup must return the original IP. Google also publishes IP ranges as JSONVerifying Google crawlers: common-crawlers.json (Googlebot), special-crawlers.json, user-triggered-fetchers.json, user-triggered-fetchers-google.json, user-triggered-agents.json
OpenAIPublished IP list per botopenai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json, openai.com/adsbot.json
AnthropicPublished IP list. Anthropic says a source IP on the list “indicates that the crawler is coming from Anthropic”claude.com/crawling/bots.json
PerplexityUser-agent match plus IP match. Perplexity recommends combining bothperplexity.ai/perplexitybot.json, perplexity.ai/perplexity-user.json

The JSON files share one shape: a creationTime and a prefixes array of ipv4Prefix or ipv6Prefix entries in CIDR notation. They change. Fetch them on a schedule rather than copying them once.

For Google, the manual check is two commands:

host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

For the published lists, pull the IPs that claim a token, then test each one against the operator’s prefixes. This example checks OAI-SearchBot; swap the token and the file for the others.

grep "OAI-SearchBot" access.log | awk '{print $1}' | sort -u > claimed.txt
curl -s https://openai.com/searchbot.json > ranges.json

python3 - <<'PY'
import ipaddress, json
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
        for p in json.load(open("ranges.json"))["prefixes"]]
for ip in open("claimed.txt").read().split():
    a = ipaddress.ip_address(ip)
    print(ip, "verified" if any(a in n for n in nets) else "NOT IN PUBLISHED RANGES")
PY

The first field of a combined-format log is the client IP. Behind a CDN or load balancer it is the proxy’s address, so log the forwarded client IP (for example, CF-Connecting-IP on Cloudflare) or the check verifies your CDN.

What to look for once the IPs are sorted:

  • Spoofed traffic. Requests carrying a bot’s name from addresses outside its list are not that bot. Treat them as ordinary scrapers. Do not count them in any report about AI crawling.
  • Verified bots that were refused. A verified OAI-SearchBot, Claude-SearchBot, or PerplexityBot getting 403 or 429 means an edge rule or rate limit is overriding your robots.txt.
  • Verified bots that ignored a Disallow. If a verified search or training crawler fetches a disallowed path more than a day after the change, record it with timestamps. That is the evidence an operator or a CDN can act on.
  • Which pages user-triggered fetchers load. ChatGPT-User, Claude-User, and Perplexity-User requests follow user questions. The URLs they fetch most are the ones being pulled into answers.

A 200 shows the bot got a response, not that it read your content. Most AI crawlers do not run JavaScript, so a client-rendered page reaches them as an empty shell. What AI crawlers see on a JavaScript site covers the test.

You will never see Google-Extended or Applebot-Extended in logs. They have no user agent. Their effect is on use, not on crawling. For tracking whether the policy changed your citations, see how to measure AI visibility.

What are the common robots.txt mistakes with AI bots?

Most damage comes from defaults nobody chose and from rules that do not mean what they look like.

  • A staging file shipped to production. User-agent: * with Disallow: / blocks every compliant crawler, Googlebot included. Check robots.txt after every deploy and every migration.
  • A template or plugin default. Themes, CMS plugins, and security tools sometimes add AI-bot blocks. Read the live file at https://yourdomain/robots.txt, not the one in the repository.
  • Assuming the * group still applies. Google’s spec says a bot follows only its most specific group, and that specific groups are not combined with *. If you add User-agent: GPTBot with Disallow: /private/, GPTBot ignores every rule in your * group.
  • Wildcards in the user-agent line. Product tokens are letters, underscores, and hyphens. User-agent: *bot* matches nothing useful. Wildcards (*, $) work only in paths, and paths are case-sensitive.
  • Using robots.txt to remove a page. Google states that for noindex to work, the page “must not be blocked by a robots.txt file.” Block the crawl and the crawler never sees the noindex, so the URL can still appear. Use noindex to remove, robots.txt to stop crawling, never both on the same URL.
  • Blocking Googlebot to avoid AI Overviews. It removes you from Google Search. Use snippet controls if you must limit AI Overviews, and accept the cost to classic results.
  • One robots.txt for many hosts. The file applies per host, protocol, and port. blog.example.com and docs.example.com each need their own. Anthropic repeats this for its bots.
  • A 5xx on robots.txt. Under RFC 9309, a server error means full disallow. An unstable robots.txt endpoint is an intermittent block.
  • Blocking by IP to opt out. Anthropic notes this can fail, because the bot cannot read robots.txt to learn your preference.

A robots.txt policy is one input. An llms.txt file can point allowed bots at the pages you want quoted, but it does not override a Disallow.

Sources

Bring us the domain.

We will tell you where you are absent, which page is in the way, and what has to be true before a citation is possible.

Request an audit