The short version: According to OpenAI's crawler documentation, blocking GPTBot opts you out of model training, not out of ChatGPT search. A separate bot, OAI-SearchBot, controls that. Anthropic and Perplexity split their bots the same way. Yet when we read the robots.txt files of the top 5,000 websites on 12 October 2026, 230 of the 446 sites whose robots.txt blocks GPTBot (52%) also block OAI-SearchBot.

Here's a familiar scenario. Someone added User-agent: GPTBot / Disallow: / to keep your docs out of training. Now a colleague asks whether that line is why ChatGPT never cites your pricing page. By OpenAI's account, it isn't. But the same file may block ChatGPT's search bot too.

That matters because assistants lean on your own site: in an internal Genezio research, company websites received 64–86% of the pages cited in answers about companies.

Our evidence has two parts. What each rule does comes from the vendors' documentation, read on 12 October 2026. What sites actually block comes from our census of 2,691 robots.txt files. Neither part measures how a block changes citations or traffic.

If you block GPTBot, do you disappear from ChatGPT answers?

No, according to OpenAI. Its documentation says: "Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training."

Each vendor runs bots for separate jobs. A training crawler collects pages for future models. A search crawler builds the index the assistant searches when it answers. A user fetcher opens a specific page because a user asked about it. Each job has its own robots.txt token, the name you put after User-agent:.

Vendor Token Job What blocking it does, per the vendor
OpenAI GPTBot Training Content "should not be used" to train foundation models
OpenAI OAI-SearchBot Search Site "will not be shown in ChatGPT search answers, though can still appear as navigational links"
OpenAI ChatGPT-User User fetch robots.txt rules "may not apply"
Anthropic ClaudeBot Training The site's "future materials" are excluded from training
Anthropic Claude-SearchBot Search "May reduce" the site's visibility in search results
Anthropic Claude-User User fetch Stops retrieval "in response to a user query"
Perplexity PerplexityBot Search Not used for model training; Perplexity recommends allowing it to appear in its search results
Perplexity Perplexity-User User fetch "Generally ignores robots.txt rules"

Table: for OpenAI, Anthropic and Perplexity, the search bot governs search answers; the training bot is a separate switch.

Note the wording: OpenAI says a blocked site "will not be shown", while Anthropic only says a block "may reduce" visibility. Google draws its line in a different place.

Does Google work the same way for AI Overviews?

No. Google has no separate token for its AI search features. Its AI features documentation says that to appear in AI Overviews or AI Mode, "a page must be indexed and eligible to be shown in Google Search with a snippet". It adds: "There are no additional technical requirements."

So the controls are the ordinary Search ones: Googlebot rules in robots.txt, plus nosnippet, max-snippet and noindex.

The Google-Extended token doesn't touch any of this. Google's crawler documentation says it "does not impact a site's inclusion in Google Search". It controls training of future Gemini models and grounding (feeding Search results to the model at prompt time) in the Gemini app and Vertex AI, Google Cloud's AI platform.

What this means: for OpenAI, Anthropic and Perplexity, the line runs between training bots and search bots. For Google, it runs between Search, which includes AI Overviews, and Gemini. Blocking Google-Extended keeps you out of Gemini grounding, not out of AI Overviews.

The vendors built independent switches. Do sites use them that way?

What do top sites actually block? Training and search together, about half the time

We fetched robots.txt from the top 5,000 domains in the Tranco ranking and got a usable file from 2,691 of them. A bot counts as "blocked" when the rule that applies to it disallows the homepage path /.

Token Job Sites blocking it Share of 2,691
GPTBot OpenAI training 446 16.6%
OAI-SearchBot OpenAI search 233 8.7%
ClaudeBot Anthropic training 425 15.8%
Claude-SearchBot Anthropic search 237 8.8%
PerplexityBot Perplexity search 332 12.3%
Google-Extended Gemini training and grounding 370 13.7%
Googlebot Google Search 79 2.9%

Table: sites block OpenAI's and Anthropic's training bots about twice as often as their search bots, but each AI search bot is still blocked by 8.7–12.3% of sites.

Taken together, 344 sites (12.8%) block at least one of the three AI search bots.

The overlap is the surprising part. Of the 446 sites blocking GPTBot, 230 (52%) also block OAI-SearchBot. The other 216 block training only. Anthropic's bots show the same pattern: 235 of the 425 sites blocking ClaudeBot also block Claude-SearchBot.

The training-only opt-out does exist in practice. 203 sites block at least one training bot while leaving all three AI search bots allowed. (We counted GPTBot, ClaudeBot, Google-Extended and CCBot, the crawler of Common Crawl, as training bots.) Templates can make the same split: Cloudflare's optional managed robots.txt disallows training bots such as GPTBot, ClaudeBot and Google-Extended, but doesn't list OAI-SearchBot, Claude-SearchBot or PerplexityBot.

The implication: robots.txt shows rules, not intent. Whether your search-bot block is a choice or inherited from a "block all AI" template, check that it matches what you want.

Some of those sites never named OAI-SearchBot at all.

How can a site block AI search bots without naming any?

Through the catch-all group. Under the robots.txt standard (RFC 9309), a bot that isn't named in any group follows the User-agent: * group. If that group disallows everything, every unnamed bot is blocked.

Of the 233 sites blocking OAI-SearchBot, 103 never mention it by name. Their catch-all rule does the blocking. Of those 103, 32 name Googlebot and allow it. That's an allowlist, which looks like this:

User-agent: Googlebot
Allow: /

User-agent: *
Disallow: /

An allowlist shuts out every AI search bot that isn't on it, including ones launched after the file was written. Sites in this group include facebook.com, x.com and imdb.com.

Named rules and catch-alls are both visible in robots.txt. Some things aren't.

What can't robots.txt show or control?

Blocks at the CDN or firewall don't appear in robots.txt, so our census can't see them. Cloudflare's documentation says robots.txt "expresses your preferences, but it does not prevent crawlers from accessing your content at a technical level", and adds that "robots.txt compliance is voluntary." Perplexity tells firewall users they "may need to explicitly whitelist" its bots. And a training block only looks forward: Anthropic says it covers the site's "future materials", not what a model has already learned.

Finally, we found no study that measures how a site's AI citations change after it blocks a search bot. The effects described here are what vendors state, not measured results.

Playbook: set each AI bot rule on purpose

  1. Decide by purpose, not by vendor: training, search index and user fetch are separate choices.
  2. Name every AI token you care about in robots.txt with the rule you intend. Don't rely on User-agent: * to decide for you.
  3. If you run an allowlist, add OAI-SearchBot, Claude-SearchBot and PerplexityBot if you want to appear in those assistants' answers.
  4. For Google AI Overviews, check Googlebot access and your snippet controls. Google-Extended won't affect them.
  5. Check your CDN and firewall bot settings, which robots.txt won't show. Verify bots against the IP ranges vendors publish.
  6. Re-check after a change. OpenAI says about 24 hours and Perplexity up to 24 hours; Google says recrawling can take "several days to several months".
  7. Confirm in your server logs that the search bots actually fetch your key pages (our JavaScript tests show how to spot them).

Methodology

  • Site list: top 5,000 domains from the Tranco list created on 11 October 2026, which averages several popularity rankings over 30 days.
  • Fetch: on 12 October 2026 at about 14:15 UTC, we requested https://<domain>/robots.txt, trying the www. host on connection errors. One attempt per host, 12-second timeout, identified non-browser user agent.
  • Sample: we counted only successful responses that weren't HTML pages: 2,691 of 5,000 domains. Many of the rest are infrastructure domains or refused our request.
  • Parsing: we followed RFC 9309. A bot's rules are the groups that name its token, case-insensitively; if none does, the * group applies. A bot is "blocked" when the most specific rule matching / is a Disallow; Allow wins ties.
  • Scope: homepage path only, on one day. The figures describe top sites that served a robots.txt, not the whole web.
  • Cross-check: an independent September 2026 census by Reese Calder of the same top 5,000 also found 8.7% of files blocking OAI-SearchBot. There, 44.5% of GPTBot blockers also blocked it.

To see which of your pages AI assistants actually cite once the bots can reach them, Genezio runs persona-based, multi-turn conversations with assistants and traces the sources behind their answers.