Key takeaways
- A user-agent string is text the visitor types. Anyone can send "GPTBot" or "Googlebot" and your log will happily record it.
- Published measurements of spoofing vary a lot by site and sample: 5.7% of AI-crawler-labelled requests across 16 crawlers in HUMAN Security's data, 16.7% for ChatGPT-User, and 80% of GPTBot claims on one small site's logs.
- Verification means checking the source IP against the operator's published ranges or a reverse and forward DNS lookup. Google, Bing, Apple and Common Crawl document both. OpenAI, Anthropic, Perplexity and Amazon publish IP lists. Meta documents neither.
- A verified visit proves a crawler fetched a page. It does not prove the page was cited or recommended.
- If you make decisions from AI crawler counts, filter to verified traffic first, or you are measuring attackers.
The first time you check your logs, it looks alarming
Phillip Stemann described it well after he first searched his server logs for AI crawlers: hundreds of requests from GPTBot, ClaudeBot and PerplexityBot, most ending in errors. It looked like AI companies were hammering his site and getting nothing.
Then he checked the IP addresses. In his week of nginx logs (8 to 15 October 2026, 69,015 requests), 80% of requests calling themselves GPTBot were fake, and for Perplexity-User it was 100%. The fakes were not reading content. They were probing for .env files, cloud credentials and Kubernetes tokens, which is what vulnerability scanners do when they want a trusted name on the door.
That is one small site, so do not treat 80% as a benchmark. But the direction matters. If you count user agents and call it AI crawler activity, you are counting a mix of real bots and anyone who borrowed their name.
Why the user-agent string proves nothing
Every request carries a user-agent string. Browsers name the browser. Crawlers name the bot, built around a token like OAI-SearchBot or PerplexityBot. It is plain text supplied by the sender.
Bing says outright that user agent strings are easy to spoof. Common Crawl says it is aware of crawlers falsely identifying themselves as CCBot. Google documents verification precisely because of "spammers or other troublemakers" claiming to be Google.
The IP address is harder to fake, because the response has to travel back to it. That is why every documented verification method starts from the IP in your log, not the name.
How much spoofing is out there?
The numbers differ by who measured and how, so read them as a range rather than a single truth.
| Source | What they measured | Spoofed share |
|---|---|---|
| HUMAN Security (via Technology Checker's Q3 2026 crawler report) | Requests with an AI crawler user agent, 16 crawlers | 5.7% average, 16.7% for ChatGPT-User |
| Centinel Analytica | News and media sites, requests naming a well-known crawler | 1 in 7 "Googlebot" requests failed verification; 84% of GPTBot claims were fake |
| ROI and Shine | One new test site, 43,345 requests over 26 days | 4.3% of declared bot traffic failed verification; 18% of "Googlebot" requests spoofed |
| Phillip Stemann | One site, one week in October 2026 | 80% of GPTBot claims, 100% of Perplexity-User claims |
The spread is large and that is fine. Different sites attract different attackers, and different vendors set different thresholds for "fraudulent." Technology Checker makes the same point: a 5.7% average and a 2.4% peak rate from two vendors are not in conflict, and neither is the correct figure for your site. What holds across all of them is the pattern. Fakers crowd around the names people trust, Googlebot first, then GPTBot and ChatGPT-User.
Centinel also found a nasty consequence for bot rules. A common setup allows search and AI crawlers by user-agent string so blocking never costs visibility. That lets the fakes straight in. In Centinel's data, 99.99% of verified requests came from the published ranges and 98% of unverified requests came from outside them, so the IP check cleanly separates the two groups.
Three kinds of AI bot, and why it changes what you read
Each AI company runs several bots for different jobs. Mixing them up leads to wrong conclusions about blocking and visibility.
| Job | OpenAI | Anthropic | Perplexity | What it means for you |
|---|---|---|---|---|
| Training | GPTBot | ClaudeBot | none | Your content may end up in future models |
| Search index | OAI-SearchBot | Claude-SearchBot | PerplexityBot | Whether you can appear in AI search answers at all |
| Live fetch for a user | ChatGPT-User | Claude-User | Perplexity-User | Someone asked a question and the AI is reading your page now |
The live-fetch row is the interesting one for GEO. Stemann saw real ChatGPT-User fetch his pages 56 times in a week, yet only 2 visits came from ChatGPT in analytics. Referral data shows you the 2. Crawler logs show you the 56.
How to verify each vendor's crawler
The eSEOspace guide to verifying AI bots lays out each vendor's documented method as of October 2026. Here is the short version.
| Operator | Reverse DNS | Published IP list | Notes |
|---|---|---|---|
| Yes (googlebot.com, google.com) | common-crawlers.json, special-crawlers.json | Common crawlers always respect robots.txt; special-case ones may not | |
| Bing | Yes (search.msn.com) | Yes | Refresh the list daily |
| Apple | Yes (*.applebot.apple.com) | Yes, CIDR prefixes | |
| Common Crawl | Yes (crawl.commoncrawl.org; no IPv6) | Yes, JSON | |
| OpenAI | No pattern published | One file per agent | Each file has its own creation time |
| Anthropic | No pattern published | One combined list | A match tells you Anthropic, not which bot |
| Perplexity | No pattern published | Two files | "Updated regularly" |
| Amazon | No pattern published | Three pages (Amazonbot, Amzn-SearchBot, Amzn-User) | |
| Meta | None documented | None documented | Cannot be verified by a documented method |
Reverse and forward DNS
Where the operator supports it, do two lookups. Resolve the IP to a hostname, check the hostname ends in the documented domain, then resolve that hostname forward and confirm it returns the same IP. ROI and Shine used this forward-confirmed reverse DNS for Google, Bing and Apple, and checked OpenAI, Anthropic and Perplexity against their published ranges.
# Reverse lookup
host 66.249.66.1
# Forward-confirm the returned hostname
host crawl-66-249-66-1.googlebot.com
Published IP files
For OpenAI, Anthropic, Perplexity and Amazon, match the request IP against the vendor's JSON. Two rules. Fetch the file fresh each time rather than pasting ranges from an article, because lists change. And do not assume a match identifies the specific bot where the list is combined, as with Anthropic's.
A warning on going further than verification: Anthropic says blocking its IP addresses "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file." If you filter by IP at the firewall, automate list updates, because a stale copy can block the real crawler.
The Meta problem
As of October 2026, Meta's crawler page publishes no IP range file and no reverse DNS method. Its only IP advice, for the link-preview crawler, is to allow user agent strings or the IP addresses (described as more secure), without saying where those addresses are listed. So a request labelled meta-externalagent cannot be verified by anything Meta documents.
That matters because Meta's crawler footprint is growing fast. Promptwatch Data's report on Meta-WebIndexer found it went from about 2.2% of tracked AI crawler requests in mid-July 2026 to 37.8% on August 9. The catch, and the report states it plainly: that share is identified by user agent only, so it can include spoofed requests and is not IP-verified like the weekly crawler report. Quote it with that caveat.
Tokens that never appear in logs
Google-Extended and Applebot-Extended are control tokens in robots.txt, not crawlers. Google says Google-Extended has no separate user-agent string and crawling uses existing Google agents. Apple says Applebot-Extended does not crawl webpages. If a report or log summary shows visits from either, treat those lines as spoofed or mislabelled.
Why this matters for GEO, not just security
There are two separate costs of trusting unverified logs.
First, you misdiagnose access. Stemann's raw view suggested AI crawlers were failing on his site. The real ones were succeeding almost every time. Someone acting on the raw numbers might have rewritten robots.txt or changed hosting to fix a problem that did not exist.
Second, you misread demand. Before you conclude how much AI crawler traffic you get, verify a sample. Technology Checker is blunt about it: if you group your own traffic by user agent and compare it to a report of verified requests, you are comparing confirmed identities to unverified claims.
The flip side is also worth knowing. In ROI and Shine's month of logs on a fresh test site, AI fetchers requested 0% assets: no JavaScript, no CSS, no images, while classic search bots pulled 42 to 79%. Server-side HTML is the entry price. And a 821-request "Claude surge" turned out to be the author's own WordPress MCP server, which any user-agent report would have booked as AI visibility.
Cloudflare is pushing in the same direction at the network level. Its BotBase tracks how each bot uses content, and a bot caught abusing those signals loses its Verified status. Verified identity is becoming the basis for allow decisions, not a nice extra.
What a verified visit does and does not prove
A verified request shows the vendor fetched the page from its own network. That is all. It does not show which product used the content, whether it appeared in an answer, or whether anyone saw it. No IP list or log entry carries that information. A verified GPTBot hit is training access. A verified OAI-SearchBot hit means you are in the index. Neither is a citation.
To close that gap you need to join three things: verified crawls, citation data from the AI answers themselves, and visitor analytics. Logs alone answer "can the bot read me," not "does the AI recommend me."
A practical routine
- Pull a week of raw access logs and filter to requests whose user agent names an AI or search crawler.
- Check each source IP against the operator's current published JSON, or run the forward-confirmed reverse DNS lookup where documented.
- Count verified and unverified separately. The unverified bucket is a security signal, not an AI visibility one. Look at what paths it requests.
- Compare verified volume per operator with expectations. Promptwatch Data's AI crawler traffic report gives weekly IP-verified shares for June 8 to September 6, 2026, with OpenAI at 79.8% of verified requests in the week of Aug 31 to Sep 6, down from 94.8% in the week of June 8. A provider that is large in that report but near zero in your logs suggests you are blocked or unreachable for it.
- Check robots.txt, CDN and WAF rules for each provider. Bot protection can return a 403, JavaScript challenge or CAPTCHA even when robots.txt allows the crawler.
- Re-run it regularly. Operator lists change, and so do the crawlers.
- Do not hard-block by IP unless you automate list updates.
If you would rather not maintain scripts, there are tools that do this for you. DarkVisitors is a lightweight option for tracking agents and bots visiting a site.
For a fuller view that connects crawls to citations and traffic, Promptwatch logs AI crawler requests that are IP-verified against operator ranges, so spoofed user agents never enter the data. Its AI crawler logs show which pages bots read, the errors they hit and a citation rate per page, which helps you tell an access problem from a content problem. Setup runs through your CDN or host (Cloudflare, Vercel, Fastly, CloudFront and others) or the WordPress plugin.
Crawler logs are available on Professional and above, and the agency plans, so confirm current plans at promptwatch.com/pricing.
Choosing a tool for crawler data
Many AI visibility trackers measure what answers say, not what bots do on your server. They are useful but answer a different question.
| Need | Approach |
|---|---|
| One-off check of your own logs | Script plus vendor IP files |
| Ongoing verified crawler logs tied to citations | An AI visibility platform with crawler analytics such as Promptwatch |
| Seeing which bots visit, nothing more | A bot tracker such as DarkVisitors |
| Prompt-level brand mentions only | Monitoring-only trackers, which will not tell you about crawl access |
If you are comparing the wider field, the GEO software directory at bestgeosoftware.com lists options by capability.
Common mistakes
- Allowing crawlers by user-agent in your WAF. That is the exact hole spoofers use.
- Counting Google-Extended or Applebot-Extended visits. They do not exist.
- Reading a verified GPTBot hit as proof of citation.
- Copying IP ranges from a blog post. Fetch the vendor file fresh.
- Treating the unverified bucket as lost AI traffic. Look at the paths it hits. If it is asking for .env, it is a scanner.
- Trusting any single spoofing percentage as a benchmark for your site.
The bottom line
Trust the IP address, not the name. Verify with DNS where the operator documents it, use fresh published IP files where it does not, and accept that Meta currently gives you nothing to verify against. Then use verified entries for what they are good at, which is confirming access and finding blocks, and measure citations and traffic with data that is built for that.
If you want a team to run this kind of technical audit for you, 1001 SEO Media handles technical SEO and AI search work, and it is also the agency that publishes this site.