How to Block AI Crawlers Without Hurting Your SEO

You can block AI training crawlers without losing any search traffic, because search indexing and AI training are governed by separate robots.txt tokens for every major engine. Blocking GPTBot, Google-Extended, ClaudeBot and CCBot stops your content feeding AI models, but Google Search still crawls you through Googlebot, Bing through Bingbot, and none of those training tokens is a ranking signal. The one thing you cannot cleanly opt out of via robots.txt is Google AI Overviews, because those are built on the search index itself. And robots.txt only works on bots that choose to obey it, so for the ones that do not you need firewall or CDN rules on top. This guide gives you the exact tokens, a copy-paste robots.txt block, and where the approach breaks down.

Will blocking GPTBot affect my Google rankings?

No. GPTBot is OpenAI's training crawler. It has nothing to do with how Google, Bing or any search engine indexes and ranks your pages. Google Search is crawled by Googlebot; Bing by Bingbot. Neither reads OpenAI's user-agent token, and neither treats "is this site blocking GPTBot?" as a ranking input. You can disallow GPTBot today and your position in Google will not move as a result.

This is the single most common mistake publishers make. The fear is that any "blocking" in robots.txt is dangerous, so they leave everything open. In reality, the Robots Exclusion Protocol lets you target one named crawler at a time. A rule aimed at GPTBot is invisible to Googlebot. That separation is the whole basis of a safe AI-blocking strategy.

GPTBot is worth singling out because it is the most-blocked AI crawler on the web. Originality.ai's ongoing tracking of the top 1,000 sites has found roughly a quarter of them disallowing GPTBot in robots.txt, and it consistently ranks as the single most-blocked AI user-agent across large samples (Originality.ai). You are in well-trodden company, and none of those sites disappeared from Google for doing it. OpenAI confirms that webmasters can control each of its bots independently, "for instance, permitting search indexing while blocking training data collection" (OpenAI).

Which AI crawlers can I block without losing search traffic?

The safe rule is: block the pure training crawlers, and leave the discovery and citation crawlers alone. Training crawlers only make AI models better. Discovery crawlers are how AI search engines find you and send referral clicks back. Here is the map.

OpenAI splits its bots into independently controllable tokens: GPTBot, which it says "is used to crawl content that may be used in training our generative AI foundation models"; OAI-SearchBot, which "is used to surface websites in search results in ChatGPT's search features"; ChatGPT-User for live user-triggered fetches; and OAI-AdsBot for ad landing-page checks (OpenAI). Blocking GPTBot stops training while OAI-SearchBot keeps you visible and citable inside ChatGPT search. Anthropic mirrors this with ClaudeBot (training), Claude-User (live fetch) and Claude-SearchBot (search quality), each set independently, and it also honours the non-standard Crawl-delay directive (Anthropic).

CCBot deserves a specific mention. Common Crawl's public archive is a widely used training-data source for downstream models, so a single rule against CCBot cuts off an indirect training pipeline you would otherwise never see. CCBot respects robots.txt and now runs on dedicated IP ranges with reverse DNS so you can verify a logged request is the real crawler (Common Crawl).

Does blocking Google-Extended remove me from Google Search?

No, and Google says so directly. In its official crawler documentation, Google states that "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search" (Google Search Central).

Google-Extended is unusual: it has no separate HTTP user-agent and does not fetch pages itself. It is purely a robots.txt control token that governs whether your content trains Gemini and grounds Gemini apps and Vertex AI. Disallowing it is a clean, consequence-free way to keep your work out of Google's model training. Your pages carry on being crawled, indexed and ranked by Googlebot exactly as before.

There is a third Google token worth knowing: Google-CloudVertexBot, which Google says handles crawls "requested by the site owners' for building Vertex AI Agents" and also has no Search impact (Google Search Central). For a publisher who simply wants search traffic without feeding Gemini, Google-Extended is the token you want in your disallow list.

The one exception: Google AI Overviews

Here is the nuance almost every guide gets wrong. Google-Extended does not control AI Overviews or AI Mode. Those features are powered by Googlebot and the Search index, not by Google-Extended (Google Search Central). So blocking Google-Extended stops future Gemini training but does nothing to keep your pages out of the AI summary that sits on top of the results page. To remove yourself from AI Overviews entirely you would have to block Googlebot, which also removes you from ordinary Search. There is no clean robots.txt lever to opt out of AI Overviews alone.

That matters commercially, because AI Overviews are where the click erosion is happening. A Pew Research Center analysis of real browsing data found that when an AI summary appeared in Google results, users clicked a traditional search link on just 8% of those visits, against 15% when no summary appeared, roughly half the click rate. Users clicked a link inside the summary itself on only 1% of visits, and ended their browsing session entirely on 26% of pages with a summary versus 16% of pages without one (Pew Research Center). About 18% of the Google searches in that March 2025 study produced an AI summary, and 88% of those summaries cited three or more sources. The takeaway for publishers is blunt: you cannot robots.txt your way out of AI Overviews, so the practical play is to manage what you can control (training) and measure the referral value of what you cannot.

Can I block AI training bots but still be cited in ChatGPT and Perplexity?

Yes, and this is the sweet spot for an ad-funded publisher. The move is to disallow the training crawlers while explicitly allowing the discovery and citation crawlers that can send you referral traffic. In practice:

• Allow: Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, Claude-SearchBot
• Disallow: GPTBot, Google-Extended, ClaudeBot, CCBot, and other pure training crawlers such as anthropic-ai and cohere-ai

With this pattern, OpenAI can still surface your pages in ChatGPT search via OAI-SearchBot and Perplexity can still link to you via PerplexityBot, so you keep the citations and any referral clicks while GPTBot and ClaudeBot stop harvesting your content for training. Perplexity's own documentation says PerplexityBot "is designed to surface and link websites in search results on Perplexity" and recommends allowing it in robots.txt (Perplexity).

The trade-off to weigh before you go further: blocking discovery and citation bots removes you from AI-search answers and can cut potential referral traffic, whereas blocking pure training bots only stops your content improving the models. That is why blanket-blocking everything is usually the wrong call for a publisher. You want to keep the doors that send visitors open. The right decision depends on how much AI referral traffic each engine actually sends you today, which you should measure before switching anything off.

A copy-paste, SEO-safe robots.txt block

This blocks AI training only and leaves every search and citation crawler allowed:

• User-agent: GPTBot / Disallow: /
• User-agent: Google-Extended / Disallow: /
• User-agent: ClaudeBot / Disallow: /
• User-agent: CCBot / Disallow: /
• User-agent: anthropic-ai / Disallow: /
• User-agent: cohere-ai / Disallow: /

Everything not named here, including Googlebot, Bingbot, OAI-SearchBot, PerplexityBot and Claude-SearchBot, stays allowed by default. Add each token as its own User-agent block; robots.txt matches on the specific token, so there is no risk of the GPTBot rule bleeding onto Googlebot.

Is robots.txt enough to block AI crawlers?

For compliant, well-behaved crawlers, yes. Robots.txt is the correct and sufficient tool for GPTBot, Google-Extended, ClaudeBot, CCBot and PerplexityBot, all of which honour it. But robots.txt is a directive that bots choose to obey, defined by the Robots Exclusion Protocol (RFC 9309). It is not access control. There are two places it stops working.

User-triggered fetchers

When a live human asks an AI a question and the assistant fetches a page in real time, that traffic often falls outside normal robots.txt rules. OpenAI's ChatGPT-User, Anthropic's Claude-User and Perplexity-User are all triggered by a user action, and vendors state robots.txt may not apply to them the same way it does to autonomous crawlers. Perplexity is explicit: its own documentation says that because a user requested the fetch, Perplexity-User "generally ignores robots.txt rules" (Perplexity). So a disallow on Perplexity-User is not reliable. If you need to stop those fetches, you need enforcement at the network layer.

Bots that ignore the rules entirely

Some actors do not declare themselves at all. Cloudflare documented Perplexity using an undeclared stealth crawler that impersonated a generic Chrome-on-macOS browser and rotated through IPs and ASNs outside Perplexity's official range to evade robots.txt and network blocks. Cloudflare estimated that stealth crawler generated 3 to 6 million requests a day, on top of 20 to 25 million from the declared crawler, and observed the activity across tens of thousands of domains (Cloudflare). Against behaviour like that, a robots.txt line does nothing. You need WAF, CDN bot-management or firewall rules; some CDNs now offer one-click managed AI-bot blocking.

Verify before you block at the firewall

If you do move to the network layer, do not trust the user-agent string, because it can be spoofed. Verify the bot is genuine first. OpenAI, Google and Common Crawl publish official IP ranges (JSON files, plus reverse-DNS confirmation for Googlebot and CCBot), so you can confirm identity before allowing or blocking. Anthropic is the awkward case: it warns that IP-based blocking "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file" (Anthropic). In other words, if you firewall-block Anthropic's IPs, ClaudeBot may no longer be able to read the very robots.txt rule you are relying on. For Anthropic, stick to robots.txt tokens rather than IP rules.

How do I see who is actually crawling and citing my site?

You cannot see crawler activity in GA4 or Search Console. Those are human-session, JavaScript-based tools, and bot hits do not appear in them. Crawler access shows up only in raw server or edge access logs and in CDN bot analytics. If you want to know whether GPTBot or CCBot are hitting you, that is where to look.

What GA4 does show is the other half of the picture: referral traffic from AI answer engines such as ChatGPT, Perplexity and Gemini appears as referral sources. That is a different and revenue-relevant signal, because it tells you whether allowing OAI-SearchBot, PerplexityBot and Claude-SearchBot is actually returning clicks worth the exposure. Measuring current AI referral value per engine is the step that turns "block everything" fear into a rational, per-engine decision. If ChatGPT search sends you meaningful traffic, you keep OAI-SearchBot open and only block GPTBot; if it sends nothing, you have a cleaner case to lock down.

This is exactly the read a clean GA4 view is built to give you. Try Ramprt free on the live demo to see your AI referral sources by engine and what that traffic earns against your site average, with no new tracking script, just a read of the GA4 you already have. If you want the strategy behind the numbers first, see how to read AI referral traffic in GA4 and what AI Overviews are doing to publisher clicks.

A five-minute action plan

1. Add the six training tokens above to robots.txt (GPTBot, Google-Extended, ClaudeBot, CCBot, anthropic-ai, cohere-ai). Leave Googlebot, Bingbot, OAI-SearchBot, PerplexityBot and Claude-SearchBot allowed.
2. Accept that AI Overviews cannot be opted out of via robots.txt without leaving Google Search. Do not block Googlebot.
3. Check GA4 referral sources to see which AI engines send you real traffic before deciding whether to tighten discovery-bot access.
4. Check server or CDN logs to confirm which AI crawlers are actually hitting you; GA4 will not show this.
5. If you find non-compliant or stealth crawlers, or want to stop user-triggered fetchers, add WAF or CDN bot-management rules and verify bots by published IP range or reverse DNS, not by user-agent string.

Done in this order, you protect your content from AI training, keep every source of search and AI-search traffic open, and make the harder calls on evidence rather than fear.