How to Tell if AI Is Scraping Your Site (and Track It Over Time)

To tell if AI is scraping your site, read your raw server or CDN access logs, not your analytics. AI training crawlers do not run JavaScript, so they never trigger the tag that GA4 relies on. They hit your server directly. So you filter your logs for the declared AI crawler user-agents (GPTBot, ClaudeBot, PerplexityBot, CCBot and the rest), then verify each hit is genuine by checking the source IP against the vendor's published IP list, because user-agent strings can be forged. To track scraping over time, schedule a job that parses your logs, verifies by IP, and records a daily count per bot, so you can see trend lines, spot new bots, and confirm whether your robots.txt blocks are actually being obeyed.

How do I tell if AI is scraping my site?

Start by understanding why your usual analytics are blind to this. GA4 is JavaScript-based. It only fires when a page renders in a real browser and executes the measurement snippet. An AI training crawler is not a browser. It requests the raw HTML, reads it, and moves on. No JavaScript runs, so no GA4 hit is ever recorded. If you go looking for scraping in your analytics dashboard, you will find nothing, and you will wrongly conclude nothing is happening.

The ground truth lives in your server logs. Every request to your site, browser or bot, is written to a raw access log by Apache or Nginx, or captured in your CDN's log stream if you sit behind Cloudflare, Fastly or similar. Each line records the source IP, the timestamp, the URL requested, the response code, the bytes served and the user-agent string. That is where AI crawlers are visible and nowhere else.

So detection is a two-step job. First, filter the logs for the user-agents that AI vendors declare. Second, verify those hits are real by checking the IP, because a user-agent alone proves nothing. Both steps matter. The first without the second gives you a number you cannot trust.

Which AI crawler user agents should I look for?

Each major AI vendor now declares one or more crawlers with a distinct user-agent token. Match the token as a substring rather than the whole string, because vendors bump version numbers (GPTBot moved to GPTBot/1.3, for example) and a rigid exact-match filter will silently miss the new version.

One point catches people out constantly. Google-Extended is not a crawler and will never appear as its own user-agent line in your logs. Google fetches everything with Googlebot. Google-Extended is only a robots.txt token that controls whether that already-crawled content is used to train Gemini and to ground Google's AI answers. You cannot count Google-Extended hits because there are none to count. All you can do is decide, in robots.txt, whether your content feeds Gemini. Per Google's own crawler documentation, blocking Google-Extended does not affect your Search ranking or inclusion. That fact alone removes the main reason publishers hesitate to block it.

The exact tokens worth keying on today are: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, CCBot, plus Bytespider, Amazonbot, Meta-ExternalAgent, Applebot-Extended, Google-CloudVertexBot and GoogleOther. As one worked reference, OpenAI's crawler docs confirm GPTBot presents the exact string Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.3; +https://openai.com/gptbot. Because vendors do change these, treat the vendor docs, not this list, as the source of truth and re-check them periodically.

Where do I find AI crawler activity in my server logs?

If you run your own Apache or Nginx, the access log is a file on the server, typically at a path like /var/log/nginx/access.log. If you sit behind a CDN, use its log export or analytics feed, because a lot of bot traffic is answered at the edge and may never reach your origin log at all. Either way, the technique is the same: grep for the tokens, then group by IP.

A first-pass Nginx command to get hit counts per source IP looks like this:

1. Filter for the declared bots: grep -iE 'GPTBot|ClaudeBot|PerplexityBot|CCBot|Bytespider|Amazonbot|Meta-ExternalAgent|GoogleOther' access.log
2. Pull the first field, which is the source IP: | awk '{print $1}'
3. Count unique IPs: | sort | uniq -c | sort -rn

That gives you a table of how many requests came from each IP claiming to be an AI crawler. It is a useful first look, but it is not proof of anything yet, because every one of those user-agent strings could be forged. That is the whole reason for step two.

Why a user-agent string is not proof

Anyone can send any user-agent header. A scraper wanting to hide can claim to be Googlebot, or claim to be a normal Chrome browser, and your log will faithfully record whatever it was told. This is not theoretical. Cloudflare reported in 2025 that it observed Perplexity using stealth, undeclared crawlers and a generic browser user-agent to fetch content from sites that had explicitly disallowed its declared bots. So a hit that says GPTBot might not be GPTBot, and a real crawler might be hiding behind a browser string. You cannot detect the second kind by grepping for known bot names at all.

How to verify a hit is a real AI crawler

Verification means confirming the source IP genuinely belongs to the vendor. There are two accepted methods.

Method one: match the IP against the vendor's published list. Each major vendor now publishes a machine-readable JSON file of the IP ranges its crawlers use, precisely so you can verify and, if you choose, build firewall rules. OpenAI publishes separate files for GPTBot, OAI-SearchBot and ChatGPT-User (linked from its bots documentation). Anthropic and Perplexity publish equivalent IP files in their own crawler docs, and Common Crawl documents CCBot's ranges. If the source IP falls inside the vendor's published ranges and the user-agent matches, you have a verified hit. If the user-agent says GPTBot but the IP is nowhere in OpenAI's list, you have a spoof, and that is itself a finding worth logging.

Method two: forward-confirmed reverse DNS. Reverse-lookup the source IP to get a hostname, check the hostname ends in the vendor's domain (for example googlebot.com), then forward-resolve that hostname back and confirm it returns the same IP. Both directions have to agree. This defeats an attacker who controls only a reverse-DNS record but not the forward one, and it is the method Google recommends for verifying Googlebot.

One caveat worth repeating: IP or user-agent blocking alone does not guarantee a vendor treats your content as opted out of training. That is why robots.txt directives plus log verification, rather than any single control, is the reliable approach.

What is the difference between a training crawler and an answer crawler?

There are two jobs a crawler does, and telling them apart changes how you read your logs and what a spike means.

Training and indexing crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot, and Google fetching content that the Google-Extended token governs for AI use) sweep your whole site in bulk. They are building or refreshing a corpus. In your logs they look like a broad crawl: many URLs, often deep into your archive, over a sustained period. This is the activity people mean when they say AI is scraping their content.

User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) behave completely differently. They hit one page in real time, the moment a person asks an AI assistant a question about that page or a topic it decides to look up. In your logs these appear as single-URL, low-volume, spiky hits that correlate with real user interest rather than a systematic sweep. A rise in user-triggered fetches is closer to a demand signal. A rise in bulk training crawls is closer to a supply drain.

Crawling is not the same as sending you traffic

Here is the number every ad-funded publisher should sit with. Crawl volume and referral volume are two entirely separate signals, and the gap between them is enormous. Cloudflare measured Anthropic's Claude making roughly 70,900 HTML page requests for every one referral it sent back during 19 to 26 June 2025. That ratio is crawler requests for HTML divided by requests where the Referer header pointed to an Anthropic hostname, normalised to a single referral. Cloudflare notes the figure may overstate the imbalance, because traffic from Claude's native app carries no Referer header and so is not counted as a referral.

The picture shifts as these products add web search and citations, but the imbalance persists. By July 2025, per Cloudflare Radar data reported via Dataconomy, Anthropic still crawled about 38,000 pages per referral, down roughly 87 percent from about 287,000 per referral in January after Claude added web search and citations, while OpenAI's ratio was about 1,091 crawls per referral. Even at OpenAI's far friendlier ratio, you are giving away roughly a thousand page reads for each visitor sent back. For a business that monetises human pageviews with display ads, that is the real story: high crawl, near-zero referral, means your content is training models while sending you almost no traffic in return.

How do I track AI crawlers over time without reading logs daily?

Detecting scraping once is a snapshot. What you actually want is a trend line, so you can see crawl volume rising or falling, spot a brand-new bot the day it appears, and confirm whether a robots.txt block you added is working. That means a small, repeatable pipeline rather than a person squinting at logs.

The pipeline is four steps, run on a schedule (a nightly cron job is plenty):

1. Parse yesterday's logs. Pull the candidate AI-crawler lines by user-agent token, from your origin logs or your CDN's log export.
2. Verify each hit by IP. Match against the vendor JSON lists, or run forward-confirmed reverse DNS. Discard or separately flag anything that fails.
3. Write one row per bot per day. Record date, bot, verified request count, unique URLs touched, and bytes served, into a CSV or a small table.
4. Chart the series. Plot verified requests per bot over time.

Once that series exists, it answers the questions that a single grep never can. You can see a sudden crawl spike from one vendor. You can see a brand-new user-agent appear and start hammering your archive. And crucially, you can prove whether a control worked: after you add a Disallow rule for a given bot, its verified hit count should fall towards zero over the following days. If it does not, you have your answer about that bot.

Why robots.txt alone does not settle it

robots.txt is a request, not a wall. Honouring it is entirely voluntary. To block declared AI training crawlers you add per-user-agent blocks, for example User-agent: GPTBot then Disallow: /, repeated for ClaudeBot, PerplexityBot, CCBot, Bytespider, Amazonbot and Google-Extended. But adding the rule and assuming it worked is the mistake. The only way to know whether a given bot obeys yours is to keep watching your logs afterwards and confirm that its verified hit count actually drops to zero. For bots that ignore robots.txt, or for stealth crawlers that never declared themselves, the block has to move up the stack to your WAF, CDN or firewall, enforced by verified IP rather than by a polite text file the bot can ignore.

A robust monitor should therefore watch for more than the well-behaved, self-declaring bots. Because Cloudflare documented crawlers using generic browser-like user-agents to bypass blocks, the more complete design also flags high-volume, non-browser access patterns from datacentre networks that do not verify against any known-good AI IP list. Those are the ones robots.txt was never going to catch.

Where does GA4 fit in, if not for detecting scraping?

GA4 cannot see scraping, but it does capture the other half of the story: AI referrals, meaning a human clicking a link inside ChatGPT, Perplexity or Gemini through to your site. In 2026, GA4 added a native AI Assistant default channel group (Medium = ai-assistant) that classifies referrals from recognised AI assistants such as ChatGPT, Gemini and Claude. Two caveats matter. Google has not published the full list of recognised referrers, and the classification depends on the Referer header, so AI apps that strip that header land in Direct or Referral instead. To capture referrals more completely you can still add a custom channel group with a Source regex along the lines of chatgpt.com|perplexity.ai|claude.ai|gemini.google.com|copilot.microsoft.com.

The point of reporting both is the contrast. Put crawl volume from your logs (how much AI is taking, per bot, over time) next to referral volume from GA4 (how many humans AI actually sends back). That side-by-side, echoing Cloudflare's crawl-to-refer ratio, is the metric that tells an ad-funded publisher whether the trade is fair.

How this connects to your lost search traffic

Scraping and lost traffic are linked but distinct problems. Crawlers take your content in bulk. Separately, Google's AI Overviews keep users on Google instead of clicking through. The Pew Research Center found that in searches returning an AI summary, users clicked a traditional result on about 8 percent of visits, versus about 15 percent when no summary appeared, and only 1 percent clicked a link inside the summary itself. Its sample of monitored US adults ran 3,900 searches producing AI summaries in March 2025. Ahrefs, analysing 300,000 keywords, found the presence of an AI Overview correlated with a 58 percent lower average clickthrough for the top-ranking page by late 2025, with position-one CTR for informational keywords falling from 0.076 in December 2023 to 0.039 by December 2025.

So the full monitoring picture for a publisher is three panes: who is crawling you (logs), how many humans AI sends back (GA4 referrals), and what AI Overviews are doing to your search clicks (Search Console and CTR trends). If you want the second and third handled without building the plumbing yourself, try Ramprt free on your existing GA4. It reads your analytics read-only, with no new tracking script, and its AI tab shows AI referrals by engine, what AI-referred traffic earns against your site average, and the AI Overview click decoupling in one place. For the wider context, see our related guides on what AI Overviews are doing to publisher traffic and how to track AI referral traffic in GA4.

A quick checklist

• Accept that GA4 is blind to scraping. Go to your raw server or CDN logs.
• Grep for the declared tokens, matching substrings so version bumps do not slip past you.
• Verify every candidate hit by IP against the vendor JSON lists or by forward-confirmed reverse DNS. A user-agent alone is not proof.
• Separate training and indexing crawlers (bulk sweeps) from user-triggered fetchers (single-page, real-time).
• Remember Google-Extended is a robots token, not a crawler, and blocking it does not touch your Search ranking.
• Schedule a daily job that verifies and records one row per bot per day, then chart the series.
• After adding robots.txt blocks, keep watching the logs to confirm the count actually falls. For bots that ignore it, block at the WAF by verified IP.
• Report crawl volume next to GA4 referral volume. The gap is the real story.