IP-Verified vs Self-Reported AI Crawler Logs: Why Spoofed Bots Skew Your Data in 2026

A user-agent string is just text anyone can type. Here is why IP-verified AI crawler logs beat self-reported ones, what spoofing does to your numbers, and how to verify bots yourself.

Key takeaways

  • A user-agent string is self-declared. Anyone can send "GPTBot" or "Googlebot" in a request header, so counting by user-agent alone inflates and distorts your AI crawler numbers.
  • In a May 2026 server-log study by ROI & Shine, 4.3% of declared bot traffic failed verification and 18% of "Googlebot" requests were spoofed.
  • Real verification means checking the source IP against the operator's published ranges (OpenAI, Anthropic, Perplexity) or using forward-confirmed reverse DNS (Google, Bing, Apple).
  • Segment verified-only traffic before you report any AI crawler metric. Unverified numbers feed bad decisions about robots.txt, firewalls and content.
  • Verification is also a privacy question: IPs are personal data under EU law, so pseudonymise early and keep retention short.

What "self-reported" actually means

Every HTTP request carries a User-Agent header. It is a text field. The client fills it in. Nothing stops a scraper running on a cheap VPS from announcing itself as GPTBot/1.1 or ClaudeBot, and a lot of them do exactly that, because many sites give known AI bots a free pass in their robots.txt or WAF rules.

Most basic log analysis tools, and plenty of CDN dashboards, classify bots by that string. That is the self-reported approach. It is fast and it is cheap. It is also wrong in a way you can't see from inside the report, because a spoofed GPTBot hit looks identical to a real one.

The Radware write-up on this problem puts the risk bluntly: attackers only need to spoof ChatGPT's user agent and route through residential proxies to get treated as a "good AI bot", including with permissions you would never hand a stranger. Your analytics don't just get noisy. Your access policy gets weaker.

What IP verification does differently

IP-verified logging asks a second question: did this request actually come from the network the operator says it comes from?

There are two common methods.

Published IP ranges

OpenAI, Anthropic and Perplexity publish the IP ranges their crawlers use. You match the request's source IP against those lists. If a request says GPTBot and the IP is not in OpenAI's range, it is not GPTBot.

Forward-confirmed reverse DNS (FCrDNS)

Google, Bing and Apple support a DNS-based check. You reverse-resolve the IP to a hostname, confirm the hostname belongs to the operator's domain, then forward-resolve that hostname and confirm it points back to the same IP. Both directions have to match.

Both methods are hard to fake, because the attacker would have to control the operator's address space or DNS. A header is free. An IP range is not.

How much fake bot traffic is there?

More than most teams assume, and it is concentrated in the names you trust most.

The ROI & Shine study analysed 43,345 HTTP requests over 26 days in May 2026 on one freshly migrated test domain with no CDN in front. The author verified bot IPs against published OpenAI, Anthropic and Perplexity ranges, and used FCrDNS for Google, Bing and Apple. Results:

  • 4.3% of all declared bot traffic failed verification.
  • 18% of "Googlebot" requests were spoofed.
  • Bing, Apple and Perplexity verified at 100%.

That last line is interesting. Spoofing is an authority-borrowing trick, so fakers crowd around the one name everyone whitelists. The sample is small and the author says so, but the pattern matches what security vendors describe at much larger scale.

ROI & Shine server-log study showing verified vs spoofed bot traffic

The same study caught a subtler problem. A burst of 821 requests that looked like a "Claude surge" turned out to be the author's own WordPress MCP server. Plenty of analyses would have booked that as AI visibility. Verification, plus knowing your own infrastructure, is what catches it.

Similarweb's log file guide makes the same point from the SEO side: requests that claim to be known AI bots but resolve to unexpected ASNs or geographies are probably spoofed and should be treated as scraper traffic.

Similarweb guide to log file analysis for AI bot signals

How spoofed bots skew your data

The damage shows up in a few specific places.

  1. Inflated crawl counts. You see thousands of "GPTBot" hits and conclude OpenAI loves your site. Some portion is a scraper.
  2. False confidence about access. Spoofed traffic can hit pages the real bot is blocked from, so your logs suggest the bot is getting through when it is not.
  3. Wrong crawl-to-citation ratios. If the numerator includes fake requests, every per-page citation rate is understated.
  4. Bad robots.txt and WAF decisions. Teams either block too much because they see abusive volume, or allow too much because the traffic looks like a trusted partner.
  5. Misleading reports. A raw user-agent number in a board deck is not AI visibility. As one practitioner guide puts it: verify before you count, and segment verified-only or the metric is meaningless.

Verified vs self-reported: side by side

Self-reported (user-agent only)IP-verified
How a bot is identifiedHeader text the client choosesSource IP matched to published ranges or FCrDNS
Spoofing resistanceNoneHigh
Setup effortMinimal, works in any log toolNeeds current IP lists and a lookup step
Risk of inflated numbersHighLow
Safe basis for firewall allowlistsNoYes
Good forRough first lookReporting, access audits, decisions
Catches a fake GooglebotNoYes

Self-reported data is fine as a starting hypothesis. It is not fine as a conclusion.

The three bot types you need to separate

Verification matters more once you split AI bots by what they do, because the consequences of blocking differ. EdgeComet's access guide groups them like this:

CategoryWhat it doesExamplesCost of blocking
Training botsBulk-collect pages to train modelsGPTBot, ClaudeBot, CCBot, BytespiderNo immediate visibility loss
Search botsBuild the live index behind AI answersOAI-SearchBot, Claude-SearchBot, PerplexityBotDirect. You leave the citable pool
User botsFetch a page because a person just askedChatGPT-User, Claude-User, Perplexity-UserImmediate. A real person gets nothing

User bots are the group most teams have never audited, and they are exactly the group a sloppy WAF rule will catch. If you can't trust the labels in your logs, you can't tell whether you are blocking a scraper or a customer's chat session.

EdgeComet guide to checking whether AI crawlers can reach your site

How to verify AI crawlers yourself

You can do a decent job manually. Here is the workflow.

Step 1: Get raw logs

Pull access logs from your origin, CDN or host. CDN logs (Cloudflare, Fastly, CloudFront, Vercel) are usually the better source, since many bots never reach your origin if they are cached or blocked at the edge.

Step 2: Filter on declared AI user agents

Start with the names you care about: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, and the Google and Meta agents.

Step 3: Check each source IP

For OpenAI, Anthropic and Perplexity, compare against the published IP lists. For Google, Bing and Apple, run the FCrDNS check:

# reverse lookup
host 66.249.66.1
# expect a googlebot.com or google.com hostname

# forward-confirm
host crawl-66-249-66-1.googlebot.com
# must resolve back to 66.249.66.1

IP lists change, so refresh them on a schedule rather than hardcoding them once.

Step 4: Segment verified vs unverified

Keep both buckets. The unverified bucket is not garbage. It tells you who is impersonating AI bots on your site and which paths they hit.

Step 5: Look at behaviour, not just identity

Real AI fetchers behave in recognisable ways. In the ROI & Shine data, AI fetchers requested 0% assets: no JavaScript, no CSS, no images. Classic search bots pulled 42 to 79%. A "bot" that loads your stylesheet and fonts is probably not a fetcher.

Step 6: Handle IPs responsibly

IPs count as personal data under EU law. Pseudonymise early, retain only briefly and never expose raw IPs in a report. The same discipline that protects visitors is what makes the verification trustworthy.

A caveat on verification

Verification tells you a request genuinely came from an operator's network. It does not tell you the operator is behaving. Cloudflare documented Perplexity using stealth, undeclared crawlers to get around no-crawl directives, which means a bot can be real and still not announce itself honestly. IP verification removes impostors. It does not remove ambiguity about declared versus undeclared behaviour from the operators themselves.

Why the verified numbers look different

When you only count verified requests, the picture is sometimes surprising. Promptwatch Data's AI crawler traffic report counts only requests whose source IP matches the crawler operator's published ranges, so spoofed user agents are excluded by design. For June 8 to September 6, 2026, it shows OpenAI at 79.8% of verified AI crawler requests in the week of Aug 31 to Sep 6, down from 94.8% in Jun 8 to 14. That is a real shift in the mix, not an artefact of impostors.

The report also suggests a useful check: compare your own logs with that mix. If a provider holds a large share there but is near zero in your logs, you are probably blocked or unreachable for that provider. You can only make that comparison honestly if your own logs are verified too. For related movement in verified traffic, see Promptwatch's data on Meta-WebIndexer's jump to nearly 38% of tracked crawler requests between mid-July and August 9, 2026.

Tools that do the verification for you

Manual checks work for an audit. They do not scale to continuous monitoring, so most teams end up using a tool.

Promptwatch is an end-to-end AI visibility platform, and its AI crawler logs are IP-verified against each operator's published ranges, so spoofed bots are filtered out before they hit your numbers. It shows which pages bots read, the errors they hit, and the crawl-to-citation path per page, which lets you tell an access problem from a content problem. Setup runs through Cloudflare, Vercel, Fastly, CloudFront, Netlify, Akamai, a custom endpoint or the official WordPress plugin.

Favicon of Promptwatch

Promptwatch

Track and optimize your brand's visibility in AI search engines
View more
Screenshot of Promptwatch website

DarkVisitors takes a different angle, tracking AI agents, bots and LLM referrals visiting your site.

Favicon of DarkVisitors

DarkVisitors

Track AI agents, bots, and LLM referrals visiting your websi
View more
Screenshot of DarkVisitors website

If you are auditing crawl behaviour more broadly, classic crawlers and log analysers still help for the Googlebot side of the picture. Screaming Frog has a log file analyser, and Botify and Oncrawl are the enterprise options for large sites.

Favicon of Screaming Frog

Screaming Frog

Industry-leading website crawler for technical SEO audits
View more
Screenshot of Screaming Frog website
Favicon of Botify

Botify

Enterprise SEO and GEO platform with AI agents for search vi
View more
Screenshot of Botify website
Favicon of Oncrawl

Oncrawl

Enterprise technical SEO crawler built for large-scale websi
View more
Screenshot of Oncrawl website
ToolFocusVerification of bot identityBest for
PromptwatchAI visibility plus crawler logsIP-verified against operator rangesTeams who want crawl, citation and traffic in one view
DarkVisitorsAI agent and bot trackingCheck the vendor's docs for methodBot-focused visibility and robots.txt control
Screaming FrogTechnical SEO crawler and log analyserManual setupHands-on technical audits
BotifyEnterprise SEO and GEOCheck the vendor's docs for methodLarge sites with in-house technical teams
OncrawlEnterprise technical SEOCheck the vendor's docs for methodLog and crawl data joined together

Whatever you pick, ask one question first: how does it decide a request is really GPTBot? If the answer is the user-agent string, the data carries every problem described above. If you want a wider list of options, the GEO software directory at bestgeosoftware.com is a good place to compare.

For a free first pass, Promptwatch's Agentic Readiness Check shows which AI crawlers can read your site, and its AI Crawler URL Inspector tests a specific URL.

Practical rules of thumb

  • Never report a raw user-agent count as AI visibility.
  • Treat "Googlebot" as the most likely spoof target and verify it first.
  • Don't allowlist by user-agent alone in your WAF. Pair it with IP checks.
  • Check your own tooling before celebrating a surge. A 821-request spike was the author's own MCP server.
  • Expect small sites to see few or no training-crawler hits. In the ROI & Shine data, a fresh domain got zero GPTBot, ClaudeBot or CCBot requests in a month, even with an allow-all robots.txt.
  • Make sure AI fetchers can get server-side HTML. They do not render JavaScript.
  • Re-pull IP ranges regularly.

FAQ

Can I verify AI crawlers by user-agent alone?

No. The user-agent is text the client chooses. Verify by IP, never by user-agent.

Which bots are most often spoofed?

The ones that get the most trust. In the May 2026 ROI & Shine study, 18% of "Googlebot" requests were fake, while Bing, Apple and Perplexity verified at 100%. One small domain is not the whole web, but the direction is clear.

Does a failed verification mean the visitor is malicious?

Not necessarily, but it means it is not who it claims to be. Treat it as scraper traffic until you know otherwise, and keep it out of your AI visibility metrics.

Do I need a CDN to do this?

No, raw server logs work. A CDN log is often more complete, though, since it captures requests before they are cached or blocked.

Bottom line

Self-reported crawler logs tell you who claimed to visit. IP-verified logs tell you who actually did. For a quick glance the first is fine. For decisions about robots.txt, firewall rules, content priorities or reporting, only the second holds up. If you are publishing a number that starts with "AI crawlers visited us", make sure the word "verified" is true.

This guide is published by 1001 SEO Media, an agency that helps brands get found in Google and cited in AI search.

Share:

© 2026 AI Search Tools · Best AI search tools and platforms · RSS

AI Search Tools is an affiliate review site. When you click links to vendors or buy through links on our site, we may earn an affiliate commission at no extra cost to you.

AI Search Tools is a review website based on user reviews on Reddit and G2, and on publicly available information. We keep everything as up to date as possible, but pricing and features can change. Always confirm the details with the vendor before purchasing.