Key takeaways
- A user-agent string is self-declared. Anyone can send "GPTBot" or "Googlebot" in a request header, so counting by user-agent alone inflates and distorts your AI crawler numbers.
- In a May 2026 server-log study by ROI & Shine, 4.3% of declared bot traffic failed verification and 18% of "Googlebot" requests were spoofed.
- Real verification means checking the source IP against the operator's published ranges (OpenAI, Anthropic, Perplexity) or using forward-confirmed reverse DNS (Google, Bing, Apple).
- Segment verified-only traffic before you report any AI crawler metric. Unverified numbers feed bad decisions about robots.txt, firewalls and content.
- Verification is also a privacy question: IPs are personal data under EU law, so pseudonymise early and keep retention short.
What "self-reported" actually means
Every HTTP request carries a User-Agent header. It is a text field. The client fills it in. Nothing stops a scraper running on a cheap VPS from announcing itself as GPTBot/1.1 or ClaudeBot, and a lot of them do exactly that, because many sites give known AI bots a free pass in their robots.txt or WAF rules.
Most basic log analysis tools, and plenty of CDN dashboards, classify bots by that string. That is the self-reported approach. It is fast and it is cheap. It is also wrong in a way you can't see from inside the report, because a spoofed GPTBot hit looks identical to a real one.
The Radware write-up on this problem puts the risk bluntly: attackers only need to spoof ChatGPT's user agent and route through residential proxies to get treated as a "good AI bot", including with permissions you would never hand a stranger. Your analytics don't just get noisy. Your access policy gets weaker.
What IP verification does differently
IP-verified logging asks a second question: did this request actually come from the network the operator says it comes from?
There are two common methods.
Published IP ranges
OpenAI, Anthropic and Perplexity publish the IP ranges their crawlers use. You match the request's source IP against those lists. If a request says GPTBot and the IP is not in OpenAI's range, it is not GPTBot.
Forward-confirmed reverse DNS (FCrDNS)
Google, Bing and Apple support a DNS-based check. You reverse-resolve the IP to a hostname, confirm the hostname belongs to the operator's domain, then forward-resolve that hostname and confirm it points back to the same IP. Both directions have to match.
Both methods are hard to fake, because the attacker would have to control the operator's address space or DNS. A header is free. An IP range is not.
How much fake bot traffic is there?
More than most teams assume, and it is concentrated in the names you trust most.
The ROI & Shine study analysed 43,345 HTTP requests over 26 days in May 2026 on one freshly migrated test domain with no CDN in front. The author verified bot IPs against published OpenAI, Anthropic and Perplexity ranges, and used FCrDNS for Google, Bing and Apple. Results:
- 4.3% of all declared bot traffic failed verification.
- 18% of "Googlebot" requests were spoofed.
- Bing, Apple and Perplexity verified at 100%.
That last line is interesting. Spoofing is an authority-borrowing trick, so fakers crowd around the one name everyone whitelists. The sample is small and the author says so, but the pattern matches what security vendors describe at much larger scale.

The same study caught a subtler problem. A burst of 821 requests that looked like a "Claude surge" turned out to be the author's own WordPress MCP server. Plenty of analyses would have booked that as AI visibility. Verification, plus knowing your own infrastructure, is what catches it.
Similarweb's log file guide makes the same point from the SEO side: requests that claim to be known AI bots but resolve to unexpected ASNs or geographies are probably spoofed and should be treated as scraper traffic.

How spoofed bots skew your data
The damage shows up in a few specific places.
- Inflated crawl counts. You see thousands of "GPTBot" hits and conclude OpenAI loves your site. Some portion is a scraper.
- False confidence about access. Spoofed traffic can hit pages the real bot is blocked from, so your logs suggest the bot is getting through when it is not.
- Wrong crawl-to-citation ratios. If the numerator includes fake requests, every per-page citation rate is understated.
- Bad robots.txt and WAF decisions. Teams either block too much because they see abusive volume, or allow too much because the traffic looks like a trusted partner.
- Misleading reports. A raw user-agent number in a board deck is not AI visibility. As one practitioner guide puts it: verify before you count, and segment verified-only or the metric is meaningless.
Verified vs self-reported: side by side
| Self-reported (user-agent only) | IP-verified | |
|---|---|---|
| How a bot is identified | Header text the client chooses | Source IP matched to published ranges or FCrDNS |
| Spoofing resistance | None | High |
| Setup effort | Minimal, works in any log tool | Needs current IP lists and a lookup step |
| Risk of inflated numbers | High | Low |
| Safe basis for firewall allowlists | No | Yes |
| Good for | Rough first look | Reporting, access audits, decisions |
| Catches a fake Googlebot | No | Yes |
Self-reported data is fine as a starting hypothesis. It is not fine as a conclusion.
The three bot types you need to separate
Verification matters more once you split AI bots by what they do, because the consequences of blocking differ. EdgeComet's access guide groups them like this:
| Category | What it does | Examples | Cost of blocking |
|---|---|---|---|
| Training bots | Bulk-collect pages to train models | GPTBot, ClaudeBot, CCBot, Bytespider | No immediate visibility loss |
| Search bots | Build the live index behind AI answers | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Direct. You leave the citable pool |
| User bots | Fetch a page because a person just asked | ChatGPT-User, Claude-User, Perplexity-User | Immediate. A real person gets nothing |
User bots are the group most teams have never audited, and they are exactly the group a sloppy WAF rule will catch. If you can't trust the labels in your logs, you can't tell whether you are blocking a scraper or a customer's chat session.

How to verify AI crawlers yourself
You can do a decent job manually. Here is the workflow.
Step 1: Get raw logs
Pull access logs from your origin, CDN or host. CDN logs (Cloudflare, Fastly, CloudFront, Vercel) are usually the better source, since many bots never reach your origin if they are cached or blocked at the edge.
Step 2: Filter on declared AI user agents
Start with the names you care about: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, and the Google and Meta agents.
Step 3: Check each source IP
For OpenAI, Anthropic and Perplexity, compare against the published IP lists. For Google, Bing and Apple, run the FCrDNS check:
# reverse lookup
host 66.249.66.1
# expect a googlebot.com or google.com hostname
# forward-confirm
host crawl-66-249-66-1.googlebot.com
# must resolve back to 66.249.66.1
IP lists change, so refresh them on a schedule rather than hardcoding them once.
Step 4: Segment verified vs unverified
Keep both buckets. The unverified bucket is not garbage. It tells you who is impersonating AI bots on your site and which paths they hit.
Step 5: Look at behaviour, not just identity
Real AI fetchers behave in recognisable ways. In the ROI & Shine data, AI fetchers requested 0% assets: no JavaScript, no CSS, no images. Classic search bots pulled 42 to 79%. A "bot" that loads your stylesheet and fonts is probably not a fetcher.
Step 6: Handle IPs responsibly
IPs count as personal data under EU law. Pseudonymise early, retain only briefly and never expose raw IPs in a report. The same discipline that protects visitors is what makes the verification trustworthy.
A caveat on verification
Verification tells you a request genuinely came from an operator's network. It does not tell you the operator is behaving. Cloudflare documented Perplexity using stealth, undeclared crawlers to get around no-crawl directives, which means a bot can be real and still not announce itself honestly. IP verification removes impostors. It does not remove ambiguity about declared versus undeclared behaviour from the operators themselves.
Why the verified numbers look different
When you only count verified requests, the picture is sometimes surprising. Promptwatch Data's AI crawler traffic report counts only requests whose source IP matches the crawler operator's published ranges, so spoofed user agents are excluded by design. For June 8 to September 6, 2026, it shows OpenAI at 79.8% of verified AI crawler requests in the week of Aug 31 to Sep 6, down from 94.8% in Jun 8 to 14. That is a real shift in the mix, not an artefact of impostors.
The report also suggests a useful check: compare your own logs with that mix. If a provider holds a large share there but is near zero in your logs, you are probably blocked or unreachable for that provider. You can only make that comparison honestly if your own logs are verified too. For related movement in verified traffic, see Promptwatch's data on Meta-WebIndexer's jump to nearly 38% of tracked crawler requests between mid-July and August 9, 2026.
Tools that do the verification for you
Manual checks work for an audit. They do not scale to continuous monitoring, so most teams end up using a tool.
Promptwatch is an end-to-end AI visibility platform, and its AI crawler logs are IP-verified against each operator's published ranges, so spoofed bots are filtered out before they hit your numbers. It shows which pages bots read, the errors they hit, and the crawl-to-citation path per page, which lets you tell an access problem from a content problem. Setup runs through Cloudflare, Vercel, Fastly, CloudFront, Netlify, Akamai, a custom endpoint or the official WordPress plugin.

DarkVisitors takes a different angle, tracking AI agents, bots and LLM referrals visiting your site.

If you are auditing crawl behaviour more broadly, classic crawlers and log analysers still help for the Googlebot side of the picture. Screaming Frog has a log file analyser, and Botify and Oncrawl are the enterprise options for large sites.

| Tool | Focus | Verification of bot identity | Best for |
|---|---|---|---|
| Promptwatch | AI visibility plus crawler logs | IP-verified against operator ranges | Teams who want crawl, citation and traffic in one view |
| DarkVisitors | AI agent and bot tracking | Check the vendor's docs for method | Bot-focused visibility and robots.txt control |
| Screaming Frog | Technical SEO crawler and log analyser | Manual setup | Hands-on technical audits |
| Botify | Enterprise SEO and GEO | Check the vendor's docs for method | Large sites with in-house technical teams |
| Oncrawl | Enterprise technical SEO | Check the vendor's docs for method | Log and crawl data joined together |
Whatever you pick, ask one question first: how does it decide a request is really GPTBot? If the answer is the user-agent string, the data carries every problem described above. If you want a wider list of options, the GEO software directory at bestgeosoftware.com is a good place to compare.
For a free first pass, Promptwatch's Agentic Readiness Check shows which AI crawlers can read your site, and its AI Crawler URL Inspector tests a specific URL.
Practical rules of thumb
- Never report a raw user-agent count as AI visibility.
- Treat "Googlebot" as the most likely spoof target and verify it first.
- Don't allowlist by user-agent alone in your WAF. Pair it with IP checks.
- Check your own tooling before celebrating a surge. A 821-request spike was the author's own MCP server.
- Expect small sites to see few or no training-crawler hits. In the ROI & Shine data, a fresh domain got zero GPTBot, ClaudeBot or CCBot requests in a month, even with an allow-all robots.txt.
- Make sure AI fetchers can get server-side HTML. They do not render JavaScript.
- Re-pull IP ranges regularly.
FAQ
Can I verify AI crawlers by user-agent alone?
No. The user-agent is text the client chooses. Verify by IP, never by user-agent.
Which bots are most often spoofed?
The ones that get the most trust. In the May 2026 ROI & Shine study, 18% of "Googlebot" requests were fake, while Bing, Apple and Perplexity verified at 100%. One small domain is not the whole web, but the direction is clear.
Does a failed verification mean the visitor is malicious?
Not necessarily, but it means it is not who it claims to be. Treat it as scraper traffic until you know otherwise, and keep it out of your AI visibility metrics.
Do I need a CDN to do this?
No, raw server logs work. A CDN log is often more complete, though, since it captures requests before they are cached or blocked.
Bottom line
Self-reported crawler logs tell you who claimed to visit. IP-verified logs tell you who actually did. For a quick glance the first is fine. For decisions about robots.txt, firewall rules, content priorities or reporting, only the second holds up. If you are publishing a number that starts with "AI crawlers visited us", make sure the word "verified" is true.
This guide is published by 1001 SEO Media, an agency that helps brands get found in Google and cited in AI search.

