Key takeaways
- A bot hitting your server and an AI answer citing your page are two separate events. Most tools only show you one of them.
- Crawler logs tell you whether access is the problem. Citation tracking tells you whether content is the problem. You need both, joined at the page level.
- OpenAI made up 79.8% of verified AI crawler requests in the week of Aug 31 to Sep 6, 2026, down from 94.8% in early June, according to Promptwatch Data. The mix is shifting fast, so a one-time robots.txt check goes stale.
- Most citation trackers monitor prompts and brands. Very few read your server logs. Fewer still tie a specific crawl to a specific citation.
- A practical stack: log-level crawler visibility, a citation tracker, and a site audit. Pick based on whether you need to diagnose or just report.
Why crawls and citations get confused
Here is a scenario I see constantly. A team checks their logs, spots GPTBot and ClaudeBot hitting the site, and concludes they are "in AI search." Then nobody cites them. Or the reverse: ChatGPT cites a page, but the team has no idea which bot fetched it or when.
The two events sit at different points in a chain. The Intent-Native CX breakdown of how agents interact with websites describes four layers: crawling, reading, citing, and (new in 2026) invoking tools through things like WebMCP. Crawling depends on robots.txt, accessibility, and server response times. Citing depends on claim precision, evidence, authority, and recency. Different problems, different fixes.

So the useful question is not "are bots visiting?" It is: which pages do bots read, which of those get cited, and where does the chain break?
The four failure points between crawl and citation
When a page is crawled but never cited, one of four things is usually going on.
- The bot got blocked or errored. A 403 from your WAF, a timeout, or a JavaScript-only page that renders empty to a fetcher. The log shows an attempt, not a read.
- The wrong bot visited. Training crawlers like GPTBot do not cite anything. Answer-time fetchers like ChatGPT-User and Perplexity-User are the ones that signal a live answer used your page.
- The page was read but did not match the query. Marketing fluff, buried answers, vague claims.
- The page was read and cited, but you are not tracking it. Your prompt set simply does not include the question that triggered it.
Promptwatch's AI crawler report makes the second point explicit. It separates training crawlers from answer-time fetchers, because lumping them together hides the signal you actually care about.
What the crawler data says right now
Two datasets are worth knowing before you set up any tooling. Both come from Promptwatch Data, and both count only requests whose source IP matches the operator's published ranges, so spoofed user agents are excluded.
The mix is moving
In Promptwatch's weekly AI crawler traffic report (June 8 to September 6, 2026), OpenAI's share of verified requests fell from 94.8% to 79.8%. Careful here: these are shares of the mix, not absolute volumes. A provider can gain share without gaining a single extra request. But the practical advice holds. Compare your own logs against the platform-wide mix. If a provider is big in the mix and near zero in your logs, you are probably blocked or unreachable for it.
New bots appear with little warning
Meta-WebIndexer went from roughly 2.2% of tracked AI crawler requests in mid-July to 37.8% on August 9, 2026, per Promptwatch Data. It briefly became the heaviest AI crawler in their logs. The link to Meta building its own search index comes from a social post and is unconfirmed by Meta, so treat the motive as alleged. The log behavior is not alleged though. If your robots.txt and CDN rules were written in 2025, you probably have no policy for it.
Claude is the smaller but fast-growing case. Its citation crawler grew from about 30 visits a day in December 2025 to several thousand a day by mid-April 2026, more than 100x, yet it was still around 1% of tracked citation crawler traffic (Claude citation crawler report). Small share, real growth, and easy to block by accident.
User agents worth checking in your logs
| Provider | User agents to check |
|---|---|
| OpenAI | GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot |
| Anthropic | ClaudeBot, Claude-User, Claude-SearchBot, claude-web |
| Google-Extended, Google-Agent | |
| Perplexity | PerplexityBot, Perplexity-User |
| Meta | meta-webindexer, meta-externalagent, meta-externalfetcher, FacebookBot |
| xAI | GrokBot, xAI-Grok, Grok-DeepSearch |
| Mistral | MistralAI-User, MistralAI-Index |
| Others | cohere-ai, DeepSeekBot |
A quick robots.txt directive for the answer-time fetchers looks like this:
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Perplexity-User
Allow: /
Remember that robots.txt is only half of it. CDN and WAF rules block bots too, and that is where most accidental blocks live.
Three tool categories, and what each can actually tell you
The research roundups (Slate, Topify, Rankscope, NovaStacks) mostly compare citation trackers. That is useful, but it skips a distinction that matters for this specific question.
| Category | Answers | Reads your logs? | Example tools |
|---|---|---|---|
| Prompt and citation trackers | Which URLs do AI engines cite for my prompts? | No | Peec AI, Otterly.AI, Profound, Rankscale |
| Bot and agent log tools | Which bots hit my site, and what did they get? | Yes | DarkVisitors, server log analyzers |
| Joined platforms | Did this crawl lead to a citation, and did that send traffic? | Yes | Promptwatch |
If you only have a citation tracker, you can see the outcome but not the cause. If you only have a log tool, you can see the cause but not whether it paid off. The gap between them is exactly what this guide is about.
Citation trackers: the outcome side
These tools run prompts across ChatGPT, Perplexity, Gemini, and AI Overviews, then report which URLs get cited. The good ones split citations into your pages, competitor pages, and third-party sites, which the Rankscope roundup calls "Citations by Source." The better ones also separate citations from mentions, since being named in an answer and being linked as a source are different wins.
Peec AI is a clean, multi-language option for tracking prompts and sources.
Otterly.AI is popular for affordable monitoring.

Profound is the heavier enterprise pick for multi-engine tracking.
Rankscale covers ranking and visibility reporting for GEO work.
Slate ties citation tracking to a content workflow.
The Slate comparison of 11 citation tracking tools is a reasonable place to start if you want a vendor-written overview, though read any roundup published by a tool vendor with that in mind.

My honest take: these tools are good at telling you that you are or are not cited. They are weak at telling you why, because they never see your server.
Bot and log tools: the cause side
DarkVisitors tracks AI agents, bots, and LLM referrals visiting your site, and it is a quick way to see who is knocking.

Technical crawlers like Screaming Frog, Sitebulb, and Lumar simulate how a crawler sees your pages, which helps with the rendering and speed failures the Semrush guide on preparing for AI agents lists as the main blockers: access, rendering, and speed.

These are good diagnostics. They still do not tell you what happened after the fetch.
Joining the two: crawl-to-citation at the page level
Promptwatch is the one I would point to for this specific job, because it puts both halves in one view. Its AI crawler logs are IP-verified (spoofed bots filtered out), show which pages each bot reads, how often, and what errors they hit, with plain-language crawl failure explanations. Then the crawl-to-citation path shows a citation rate per page, so you can tell an access problem from a content problem without guessing.

The loop works like this in practice:
- Agent Analytics shows GPTBot, ChatGPT-User, ClaudeBot and others hitting a page, or failing to.
- Page Tracker shows whether any tracked prompt actually cites that URL.
- Visitor analytics shows whether the citation sends visits and conversions. (One wrinkle: ChatGPT stopped sending a referrer on October 8, 2026, so there is a setting that counts visits tagged utm_source=chatgpt.com as ChatGPT traffic.)
- Pages that get crawled but not cited come back as actions on the Action Board, with Content Refresh suggestions.
Setup is done through your CDN or host (Cloudflare, Vercel, Fastly, CloudFront, Netlify, Akamai and others) or the WordPress plugin, and takes minutes. Crawler logs start on the Professional plan at $245/mo, with 25M crawler logs included; Essential does not include them. Confirm current numbers at promptwatch.com/pricing. If you want to see where you stand first, the free AI visibility report and the Agentic Readiness Check cost nothing.
TreeHouse, the company behind Pararius, is a decent proof point. They fixed crawler access, published proprietary rental data, and ended up with AI search at roughly 25% of organic conversions, almost all of it from ChatGPT (case study). Access first, then content. That order is not an accident.
Other platforms that connect tracking to action include xSeek (links prompt tracking to real cited pages), bydefault and OGTool (track citations and publish), and Seology or Nuwtonic on the agentic side. I have not tested their log-level depth, so verify whether they read your server data or only work from prompts.
A practical workflow you can run this week
Step 1: baseline your logs
Pull 30 days of server or CDN logs and count requests by user agent. Separate training bots from answer-time fetchers. Save the numbers. Without a baseline you cannot tell a change from noise.
Step 2: compare against the platform mix
If OpenAI dominates the public mix but your logs show almost none of it, you have an access issue. Check robots.txt, then CDN and WAF rules, then rate limiting. WAFs are the usual culprit.
Step 3: list your crawled pages and check citations
Take the pages that fetchers actually read. For each, ask whether any AI engine cites it for a relevant prompt. Pages that are read but never cited have a content problem.
Step 4: fix the content, not just the crawl
The Intent-Native piece lists what hurts citation: vague marketing language, answers buried below the fold, and unclear page purpose. Fixes are boring and effective: lead with the direct answer, state specific claims with evidence, keep terminology consistent, update stale facts.
Content type matters too. In Promptwatch's data for Sep 28 to Oct 4, 2026, product pages were the most cited type in ChatGPT at 22.4% of citations, followed by landing pages and listicles. And content half-life data shows landing pages and docs stay cited longest while news loses half its pages within 36 days. If you publish news-style content, plan to refresh it.
Step 5: re-check monthly
The bot mix moved from 94.8% to 79.8% OpenAI in three months, and Meta appeared out of nowhere. Monthly log reviews are the minimum.
Metrics worth watching
| Metric | What it tells you |
|---|---|
| Crawls per page by bot | Whether access works |
| Error rate per bot | Where WAF or rendering breaks things |
| Citation rate per crawled page | Content problem vs access problem |
| Self-citation rate | Share of citations pointing at your own domain |
| AI-referred visits | Whether citations turn into traffic |
If you want the definitions, the Promptwatch Academy has a citations vs mentions guide and a page on attributing AI traffic.
What about the invocation layer?
The fourth layer, agents calling your site's tools through WebMCP, is real but early. Intent-Native CX scanned 50 German heating company sites in February 2026 and found 22 with interactive configurators and zero exposing them to AI. No tool I know of ties invocations to citations yet. For now, treat it as a watch item, not a budget line.
Picking a setup by situation
| Situation | Start with |
|---|---|
| Small site, just curious | Free AI visibility report plus a log check |
| Reporting to leadership on mentions | A citation tracker such as Peec AI or Otterly.AI |
| Need to diagnose why you are not cited | Log-level crawler visibility joined to citations (Promptwatch) |
| Technical SEO team auditing rendering | Screaming Frog, Sitebulb or Lumar, plus a citation tracker |
| Agency managing many clients | A platform with agency plans and pitch projects |
More options are listed in the GEO software directory at bestgeosoftware.com, and if you are weighing in-house effort against outside help, 1001 SEO Media runs GEO and technical SEO work for teams that would rather not build this themselves.
The short version
Stop treating crawl counts as a visibility metric. They tell you the door is open, nothing more. Set up verified logs, split training bots from answer-time fetchers, join the crawled pages to the citations they earn, and fix whichever side is broken. Start with a 30-day log baseline today; everything else depends on having it.






