AI Crawler List 2026: User Agents, What They Do, and What to Block

If you run an ad-funded site, the bots in your logs are no longer just Googlebot. In 2026 you are being crawled by named agents from OpenAI, Anthropic, Perplexity, Google and Common Crawl, and each of them does a different job. Some harvest your content to train models. Some build the index that sits behind AI answers and citations. Some fetch a single page live when a person asks a question. The single most important thing to understand is that these are separate systems with separate controls: blocking GPTBot does nothing to OAI-SearchBot, and disallowing Google-Extended does not remove you from AI Overviews. This guide lists every major AI crawler, the exact user agent it sends, whether it obeys robots.txt, and how to block it properly.

What are the main AI crawlers in 2026?

The major AI crawlers cluster into three functional categories, and this split is the key to handling them correctly.

• Training crawlers harvest content to train models. This group includes GPTBot (OpenAI), ClaudeBot (Anthropic), the Google-Extended control token, and Common Crawl's CCBot.
• Search and indexing crawlers build the index behind AI answers and citations. This is OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic) and PerplexityBot.
• User-triggered fetchers retrieve a page live when a person asks a question. This is ChatGPT-User (OpenAI), Claude-User (Anthropic) and Perplexity-User.

Blocking a bot in one category has no effect on the others. That matters commercially: you might want to opt out of training but stay visible in AI search, or the reverse. Because the controls are independent, you can do exactly that, but only if you target the right token.

Quick-reference table

What is the user agent for GPTBot, ClaudeBot and PerplexityBot?

These are the exact strings, taken from each vendor's official documentation. Do not guess at them, and do not rely on the string alone for blocking decisions (more on that below).

OpenAI runs four named bots:

• GPTBot: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.3; +https://openai.com/gptbot
• OAI-SearchBot: ...compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot
• ChatGPT-User: ...compatible; ChatGPT-User/1.0; +https://openai.com/bot
• OAI-AdsBot: ...compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot

The robots.txt tokens are GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot, and OpenAI states that each setting is independent. That is the important detail: you can allow OAI-SearchBot so you stay in ChatGPT search results while disallowing GPTBot to opt out of training, per the OpenAI crawler documentation. OpenAI also notes that a robots.txt change can take roughly a day to be reflected in its search systems.

Anthropic runs three bots, all of which it says honour robots.txt:

• ClaudeBot, robots.txt token ClaudeBot (training)
• Claude-SearchBot, token Claude-SearchBot (search indexing)
• Claude-User, token Claude-User (user-directed fetch)

Anthropic states these blocks operate independently, so disallowing ClaudeBot does not affect Claude-SearchBot or Claude-User. It publishes verified crawler IPs at claude.com/crawling/bots.json, supports the non-standard Crawl-delay directive, and warns that IP-based blocking may not reliably opt you out, because blocking the wrong ranges can stop Anthropic from reading your robots.txt in the first place. See the Anthropic Help Center.

Perplexity runs two:

• PerplexityBot: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot), which Perplexity says respects robots.txt
• Perplexity-User: ...compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user), which is user-initiated and, in Perplexity's own words, generally ignores robots.txt

IP lists are at perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json, documented in the Perplexity crawler docs. Treat Perplexity's compliance claims with caution: in August 2025 Cloudflare reported catching Perplexity using undeclared crawlers and rotating IPs to fetch pages that had blocked its named bot, which is covered under blocking below.

Which AI crawlers are for training and which are for live answers?

This is the distinction that decides what you are actually giving up when you block something.

Training crawlers take your content to build or refine a model. Once your page is in the training set, blocking later does not remove it from the model. GPTBot, ClaudeBot and CCBot are the main ones, plus the Google-Extended token that governs Gemini and Vertex training.

Search and indexing crawlers build the live index that AI answers cite. If you block OAI-SearchBot, Claude-SearchBot or PerplexityBot, you are removing yourself from the pool of pages those products can surface and link to. For a publisher chasing AI referral traffic and citations, this is usually the category you want to keep open.

User-triggered fetchers fire when a specific person asks a question that needs your specific page right now. ChatGPT-User, Claude-User and Perplexity-User are on-demand. They are low volume but high intent, and they are the hardest to block because they are treated as user-requested fetches rather than crawls.

The scale of each category is lopsided. Cloudflare found that over the twelve months to mid-2025, 80% of AI crawling was for training, 18% for search and just 2% for user actions, per Cloudflare's crawl-to-click gap analysis. So the overwhelming majority of AI bot load on your server is training, which is also the category that sends you nothing back.

The special case: Google-Extended and AI Overviews

Google-Extended is not a crawler. It has no user agent of its own. It is only a robots.txt product token that controls whether content already crawled by Googlebot may be used to train Gemini Apps and Vertex AI generative APIs. Google states explicitly that Google-Extended is not a ranking signal, does not affect a site's inclusion in Google Search, and is not a method for managing how content appears in Search, per the Google crawler documentation.

This is the trap that catches publishers. Disallowing Google-Extended feels like opting out of Google's AI, but AI Overviews are generated from the normal Googlebot index, so you remain in them regardless. There is currently no robots.txt lever that keeps you in Google Search while removing you from AI Overviews. The only way out of AI Overviews is out of the index entirely, which is almost never what a publisher wants.

Do AI crawlers obey robots.txt?

Some do, some do not, and the honest answer is that robots.txt is a request, not an enforcement mechanism.

The named training and indexing bots generally comply. GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, CCBot and the Google-Extended token all honour robots.txt according to their vendors. So for the biggest, noisiest category (training) a correct robots.txt genuinely reduces load and opts you out with compliant vendors.

User-triggered fetchers are where it breaks down. Perplexity-User is user-initiated and generally ignores robots.txt, which Perplexity states openly: the reasoning is that a human explicitly asked for that page, so it is treated like a browser request rather than a crawl. ChatGPT-User behaves as an on-demand fetcher too. That means robots.txt alone will never stop every AI hit.

And then there is outright evasion. In August 2025 Cloudflare published evidence that Perplexity was using undeclared crawlers, rotating IP addresses outside its published ranges and a spoofed Chrome-on-macOS user agent to fetch content from sites that had explicitly disallowed PerplexityBot, per Cloudflare's investigation. Common Crawl and others have likewise noted that bots frequently spoof their user-agent strings, so a request claiming to be CCBot or PerplexityBot may be something else entirely. A robots.txt rule only helps against agents that both identify honestly and choose to comply.

How do I block a specific AI crawler?

Blocking is a two-layer job. Layer one is robots.txt, which stops compliant training and indexing bots. Layer two is a WAF or CDN rule keyed on verified IP ranges, which stops the user-triggered fetchers and the non-compliant or spoofing crawlers that ignore robots.txt.

Layer one: robots.txt

Add a block for each token you want gone. To opt out of the main training pipelines while staying in AI search and citations:

• User-agent: GPTBot then Disallow: / (OpenAI training)
• User-agent: ClaudeBot then Disallow: / (Anthropic training)
• User-agent: CCBot then Disallow: / (Common Crawl)
• User-agent: Google-Extended then Disallow: / (Gemini and Vertex training)

Leave OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed if you want to keep appearing in AI answers. Blocking CCBot matters more than it looks: Common Crawl's public dataset has historically fed many AI training pipelines, so a single Disallow: / for CCBot closes a back door even for AI companies you never see hit your logs directly. Anthropic also supports a Crawl-delay directive if you want to slow ClaudeBot rather than block it, for example Crawl-delay: 1 under User-agent: ClaudeBot.

Layer two: verified-IP blocking at the edge

For anything that ignores robots.txt (Perplexity-User, ChatGPT-User, spoofers) you need a server, WAF or CDN rule. The robust verification pattern is two steps: first match the vendor's robots.txt user-agent token, then confirm the source IP falls inside the vendor's official published IP-range JSON, or use reverse-then-forward DNS where the vendor supports it. OpenAI, Anthropic, Perplexity and Common Crawl all publish machine-readable IP lists for exactly this, and CCBot now runs on dedicated IP ranges with reverse DNS for verification. Note that the Perplexity case shows even verified-IP blocking has limits when a vendor deliberately fetches from IPs outside its own published ranges, which is where reputable CDNs fall back on behavioural detection.

Do not allow or deny on the user-agent string alone. Because spoofing is common, UA matching without an IP check will either let impostors through or block legitimate traffic. Match the token, then verify the IP.

Worked example: allow AI search, block AI training and live fetching

1. In robots.txt, Disallow: / for GPTBot, ClaudeBot, CCBot and Google-Extended. Leave OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed.
2. At your WAF or CDN, add a deny rule for requests whose UA contains Perplexity-User/1.0 or ChatGPT-User/1.0 AND whose IP is inside perplexity.com/perplexity-user.json or openai.com/chatgpt-user.json respectively. (Verifying the IP here confirms the request is genuine before you block it, and lets you drop spoofers that fake the same string.)
3. Keep a catch-all rule that challenges or rate-limits any client claiming a known bot UA from an IP outside that vendor's published ranges.

That combination opts you out of training, keeps you in AI search, and closes most of the live-fetch and spoofing gaps that robots.txt cannot reach.

Should you block AI crawlers at all?

This is a trade-off, not a moral question, and the numbers cut both ways.

The case for blocking is that AI search demonstrably eats click-through. Pew Research found that when a Google AI summary appears, users click a traditional result link on only 8% of visits, versus 15% when there is no AI summary, roughly halving click-through, and just 1% of visits with an AI summary produced a click on a source cited inside the summary, per Pew Research Center. In the same study, users ended their browsing session on 26% of pages with an AI summary versus 16% of pages with only traditional results. Add the crawl imbalance and the picture worsens: Cloudflare recorded Anthropic crawling 286,930 pages per referred visit in January 2025, improving to 38,065 to 1 by July but still the most crawl-heavy, while OpenAI ran roughly 1,217 to 1 falling to 1,091 to 1. You are giving a lot of content for very little traffic.

The case against blocking is that the smaller stream of AI-referred visitors tends to convert and engage better than average organic traffic, so a modest number of AI referrals can be worth more per head than the raw count suggests. For many publishers the sensible middle path is exactly the worked example above: block training, which is the bulk of the load and returns nothing, but stay in AI search and citations, where the referral stream lives.

You cannot manage what you cannot measure

Before you block anything, know what AI is already sending you, because the reporting is worse than most publishers assume. On 13 May 2026 GA4 began auto-assigning an ai-assistant medium and an AI Assistant default channel. The built-in list started at three referrers (ChatGPT, Gemini, Claude) and has since expanded, but it is still incomplete: as of mid-2026 the documented set covers ChatGPT, Gemini, Copilot, Grok and DeepSeek, leaving Perplexity and others buried under Referral, Direct or Unassigned. AI apps and embedded browsers also frequently strip the referrer header, so between roughly a fifth and two fifths of AI-originated visits land as Direct. The practical fix is a custom channel group with a Source regex covering chatgpt.com, perplexity.ai, claude.ai, gemini.google.com, copilot.microsoft.com and the rest.

Search Console has its own gap. Google's new generative AI performance report, announced 3 June 2026 and rolling out to a subset of site owners, shows impressions in AI Overviews and AI Mode broken down by page, country, device and date, but with no click data, no CTR, no average position and no query-level breakdown, per the Google Search Central blog. Within one search result, a URL shown in both an AI Overview and the blue links counts as a single impression, and every link inside an AI Overview shares the single position assigned to that Overview, per Search Console help. So you can see that AI is showing your pages, but not what it costs or earns you.

This is the gap Ramprt exists to close. It reads your existing GA4 (no new tracking script, read-only) and gives you a clean AI view: referrals by engine, which crawlers are taking your content, how often AI answers are built on your pages, and what AI-referred traffic earns versus your site average. If you are about to decide what to block, see the real numbers first with the free demo at ramprt.io/demo.

For the wider picture, see our guides on measuring AI traffic in GA4 and what AI Overviews are doing to your search clicks.

FAQ

Does blocking GPTBot remove me from ChatGPT search?

No. GPTBot handles training. ChatGPT search is served by OAI-SearchBot, which is a separate bot with its own robots.txt token. OpenAI states each setting is independent, so you can disallow GPTBot and still allow OAI-SearchBot to stay in ChatGPT search.

Will disallowing Google-Extended keep me out of AI Overviews?

No, and this is the most common mistake. Google-Extended only controls Gemini and Vertex training. Google states it is not a ranking signal and does not affect Search inclusion, and AI Overviews are generated from the normal Googlebot index, so you remain in them. Blocking Google-Extended opts you out of training only.

Can I stop Perplexity with robots.txt?

Only partly. PerplexityBot, the indexing and citation crawler, is stated to respect robots.txt. Perplexity-User, the live user-driven fetcher, is user-initiated and generally ignores it, and Cloudflare has documented Perplexity using undeclared crawlers to evade blocks entirely. A hard block needs a WAF or CDN rule plus behavioural detection, not robots.txt alone.

Is matching the user-agent string enough to block a bot safely?

No. Common Crawl and Perplexity both report that bots frequently spoof their user-agent strings, and Cloudflare has caught Perplexity doing exactly that. Verify by matching the token AND confirming the source IP against the vendor's published JSON, or use reverse-then-forward DNS where supported.

Why block CCBot if I never see AI companies in my logs?

Because CCBot builds Common Crawl's public dataset, which has historically fed many AI training pipelines. Blocking CCBot with User-agent: CCBot and Disallow: / closes an indirect training route even for AI companies whose own bots never touch your site.

What is the single robots.txt setup for most publishers?

Disallow GPTBot, ClaudeBot, CCBot and Google-Extended to opt out of training, and leave OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed to stay in AI search and citations. Then add edge rules for the user-triggered fetchers if you want a hard block.