Key takeaways
- AI crawlers fetch raw HTML and do not run JavaScript, so Google Analytics and tag-based tools never see them. You need server logs or CDN/edge data.
- The most useful split is training bots (GPTBot, ClaudeBot) versus answer-time bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User). A tool that blends them into one count hides the half that produces citations.
- Three tool types exist: edge tools (Cloudflare, Vercel), standalone bot trackers (Known Agents/Dark Visitors), and crawler analytics inside AI visibility platforms (Promptwatch and others).
- I did not run a hands-on benchmark of every tool for this guide, so I give you a test plan to verify any tool on your own live site instead of invented results.
- Always check IP verification. User-agent strings are trivially spoofed.
A note on "tested on a live site"
Most roundups with that kind of title are really vendor docs rewritten. This one is built from vendor documentation, first-party data from Promptwatch Data, and a comparison published by Prefer in September 2026. Prefer ranks itself first and says it did not open accounts with the other tools it lists, so treat its rankings with the same skepticism you'd apply to any vendor-authored list. Prices below are secondhand unless noted. Confirm them on each vendor's site.
What I can give you is a way to test any of these tools against your own logs in an afternoon. That section is further down, and it matters more than the ranking.
Why normal analytics misses AI bots
Google Analytics, Plausible, and any script-based tool rely on JavaScript executing in a browser. GPTBot, ClaudeBot, and PerplexityBot request the HTML and leave. No script runs, so no hit is recorded.
That leaves two places where the evidence lives:
- Your origin server or host logs (nginx, Apache, application logs)
- Your CDN or edge layer (Cloudflare, Fastly, Vercel, CloudFront, Akamai, Netlify)
Every tool in this guide reads one or both. The differences are in what they do after that: identification, verification, separating bot purpose, and tying crawls to citations.
What the data says is worth tracking
The mix of bots hitting sites is shifting quickly, which is the best argument for watching your own logs rather than assuming.
- In Promptwatch's AI crawler traffic report (weekly verified requests, June 8 to September 6, 2026), OpenAI made up 79.8% of verified AI crawler requests in the week of Aug 31 to Sep 6, down from 94.8% in the week of Jun 8 to 14. Other providers are taking a growing share.
- Meta-WebIndexer went from about 2.2% of tracked AI crawler requests in mid-July 2026 to a peak of 37.8% on August 9, according to Promptwatch's Meta-WebIndexer data. That is roughly 17x in under a month, and it made Meta the heaviest AI crawler in their logs that day. If your infrastructure bills per request, that kind of surge shows up on an invoice.
- Claude's citation crawler grew from about 30 visits a day in December 2025 to several thousand a day by mid-April 2026, per Promptwatch's Claude crawler report. Peak daily share was 1.73%, so it is still a secondary channel. Unblock it and monitor it, but it probably shouldn't drive your whole plan yet.
The practical rule from the crawler traffic report is a good one: compare your logs to the overall mix. If a provider is a big share in the benchmark but near zero in your logs, you are probably blocked or unreachable for that provider. Check robots.txt, CDN rules, and WAF rules for each one.
The three types of tools
Edge and host tools
Cloudflare AI Crawl Control is free on every Cloudflare plan, according to Prefer's comparison. Vercel has its own bot data. They see every request at the edge, which is accurate and easy. The catch is obvious: they only see sites they serve. No Cloudflare zone, no Cloudflare data.
One gotcha worth repeating. Cloudflare's managed robots.txt block covers GPTBot, ClaudeBot, and Google-Extended but, per Prefer's write-up, not OAI-SearchBot or PerplexityBot. A plain User-agent: * disallow blocks all of them. Check the actual rule, not what you meant to configure.
Standalone bot trackers
Dark Visitors, now Known Agents, is the usual standalone pick. Per Prefer it has a free plan, 2,000+ named agents, and AI chat referral tracking, and it works on any host.

It is a good starting point if you only want to know who is visiting and don't need citation or content data.
Crawler analytics inside AI visibility platforms
Here crawler logs sit next to prompt tracking and citations, which is where it gets useful. Knowing GPTBot fetched a page is a fact. Knowing whether that page then got cited in ChatGPT answers is the insight.
Promptwatch does this with IP-verified AI crawler logs (its Agent Analytics). It only counts requests whose source IP matches the operator's published ranges, so spoofed user agents are dropped. It works with Cloudflare, Vercel, Fastly, and raw server logs, plus the WordPress plugin, and it shows the crawl-to-citation path with a citation rate per page. Crawl failures come with plain-language explanations, which saves a lot of log squinting.

The tracked list is broad: OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot), Anthropic (ClaudeBot, Claude-User, Claude-SearchBot, claude-web), Google, Perplexity, xAI, Mistral, Meta (including meta-webindexer), Cohere, and DeepSeek.
Plan note: Essential ($95/mo) has no crawler logs. Professional ($245/mo) includes 25M crawler logs, Business ($579/mo) 100M. Agency plans start at $199/mo with 10M logs. Confirm at promptwatch.com/pricing.
Other platforms in the same bucket, with prices as listed by Prefer (unverified): Otterly ($189), Qwairy ($519), and Profound (custom).

I'd check each one's docs for IP verification and the training versus answer split before paying. For more options in this category, the GEO software directory at bestgeosoftware.com is a useful place to browse.
Technical SEO crawlers are a different thing
Screaming Frog, Sitebulb, Oncrawl, JetOctopus, Botify, and Lumar crawl your site as a bot would. Some of them (Oncrawl, JetOctopus, Botify) also analyze server logs, mostly for Googlebot behavior. They are strong for crawl budget work, but check whether they name AI bots and split training from answer traffic before assuming they cover this use case.

Comparison table
| Tool | Type | Data source | Separates training vs answer bots | Price (verify) |
|---|---|---|---|---|
| Cloudflare AI Crawl Control | Edge | Cloudflare zone only | Partly, by bot category | Free on all plans (per Prefer) |
| Vercel bot data | Host | Vercel deployments only | Check docs | Included with Vercel |
| Known Agents / Dark Visitors | Standalone | Any host | Named agents, 2,000+ | Free plan (per Prefer) |
| Promptwatch | Visibility platform | Cloudflare, Vercel, Fastly, raw logs, WordPress plugin | Yes, per-crawler view, IP-verified | From $245/mo with crawler logs |
| Otterly | Visibility platform | Check docs | Check docs | $189 (per Prefer) |
| Qwairy | Visibility platform | Check docs | Check docs | $519 (per Prefer) |
| Profound | Visibility platform | Check docs | Check docs | Custom |
| Prefer | Log-based | Server logs | Yes, per its own docs | $99/mo (self-reported) |
Features that separate good from mediocre
- IP verification. Anyone can set a user agent to GPTBot. Verified tools match against the operator's published IP ranges.
- Bot purpose. Training, search indexing, and user-triggered fetch mean different things for your strategy.
- Error reporting. 403s, 429s, timeouts, and robots.txt blocks, per page.
- Crawl-to-citation link. Does the tool tell you if a crawled page ever got cited?
- Host coverage. Will it work with the CDN you already run?
- Retention and volume limits. Log-based tools bill by volume, so a surge like Meta-WebIndexer's can eat your quota.
A live-site test plan you can run yourself
This is how I'd test any tool before committing. It takes a few hours.
Step 1: Pull a raw baseline
Export seven days of raw logs from your origin or CDN. Filter with grep for the main user agents:
grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|meta-webindexer" access.log | wc -l
Note counts per bot. This is your ground truth.
Step 2: Compare against the tool
Run the tool for the same window. Counts should be close. Large gaps usually mean the tool is filtering spoofed traffic (good) or missing a bot from its list (bad). Ask which.
Step 3: Plant a spoof
Send a curl request from your own machine with a GPTBot user agent:
curl -A "GPTBot/1.1" https://yoursite.com/some-page
A tool with IP verification should not count it as genuine GPTBot. A tool that does is just reading strings.
Step 4: Check the training vs answer split
Look for whether the dashboard shows OAI-SearchBot and ChatGPT-User separately from GPTBot. If it only shows "OpenAI", you can't tell if your pages are being learned from or retrieved for answers.
Step 5: Block one bot in a staging rule
Add a robots.txt rule for a single bot, wait a day, and see whether the tool flags the change. The robots.txt generator for AI and AI crawler URL inspector are free ways to check the rule you wrote is what you intended. The AI bots directory lists 400+ crawlers with robots.txt guidance.
Step 6: Follow one page to a citation
Pick a page, confirm it was crawled, then check whether any AI answer cited it. If your tool can't connect the two, you will have to do it by hand.
Which one should you pick?
- Site on Cloudflare, just curious about bots: turn on AI Crawl Control. It's free.
- Any host, want names and referrals without spending: Known Agents.
- Want to know whether crawls become citations and fix the gaps: an AI visibility platform with crawler logs. Promptwatch is the one I'd put first here, because logs, citation tracking, site audits, and the Content Agent live in one loop, and the 7-day free trial lets you run the test plan above before paying. The free trial and the free AI visibility report are the no-risk starting points.
- Large site, log analysis is mostly about Googlebot and crawl budget: an enterprise crawler such as Botify or Oncrawl, then confirm AI bot coverage.
Common mistakes
- Reading a total "AI bots" number as meaningful. Split by purpose.
- Blocking training bots and accidentally blocking answer bots too with a blanket rule.
- Trusting user-agent strings without IP checks.
- Ignoring crawl load. If you pay by request, bot surges cost money.
- Treating Claude traffic as a priority just because it's growing. It's still a small share per Promptwatch's data.
Bottom line
Pick the tool that matches where your traffic is served and what question you need answered. If the question is just "who is hitting my site", free options cover it. If it's "why isn't AI citing me", you need crawl data tied to citations. Whichever you choose, run the spoof and baseline tests first. Five minutes with curl tells you more about a tool's accuracy than any vendor comparison page, including this one.



