Key takeaways
- AI crawler logs show which bots reach your pages and which hit errors. That turns a vague "we are invisible in ChatGPT" complaint into a ticket a developer can act on.
- Count only IP-verified requests. Spoofed user agents pollute raw logs and send developers chasing problems that do not exist.
- Rank fixes by crawl failures on pages that matter commercially, then by blocked providers, then by freshness and structure.
- Compare your logs to the market mix. If a provider is large in aggregate data but near zero on your site, you are probably blocked.
- Close the loop: after a fix ships, check the crawl, the citation and the traffic. Otherwise the ticket just disappears.
Why marketing and developers talk past each other
Marketing says "we are not showing up in ChatGPT." Developers hear something with no owner, no URL and no error code. The ticket gets parked behind a checkout bug, and nobody is wrong to park it.
The fix is not more persuasion. It is better evidence. AI crawler data is the closest thing to a server-side witness for AI search: it records which bots asked for which URLs, what status code came back, and what happened next. Developers already trust logs. They do not always trust a visibility score.
That matters more now that most marketing teams use AI heavily but struggle to act on its output. One Improvado write-up puts AI adoption among marketers at 94%, and says teams hesitate when they cannot see why a system recommended an action. Crawler logs are the opposite of a black box. Every row is a request, a URL and a response.
What AI crawler data actually tells you
AI crawlers are not one thing. Some collect training data, some build search indexes, and some fetch a page live while a user waits for an answer. Which one visits changes what a failure costs you.
Promptwatch Data tracks IP-verified requests from these user agents, grouped by provider:
- OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot
- Anthropic: ClaudeBot, Claude-User, Claude-SearchBot, claude-web
- Google: Google-Extended, Google-Agent
- Perplexity: PerplexityBot, Perplexity-User
- xAI: GrokBot, xAI-Grok, Grok-DeepSearch
- Mistral: MistralAI-User, MistralAI-Index
- Meta: meta-webindexer, meta-externalagent, meta-externalfetcher, FacebookBot
- Cohere and DeepSeek: cohere-ai, DeepSeekBot
A user-triggered fetcher like ChatGPT-User hitting a 403 is a lost answer in real time. A training crawler getting blocked is a policy decision you may have made on purpose. Those two rows deserve very different tickets, and a developer will appreciate that you know the difference.
Why verification matters
Anyone can put "GPTBot" in a user agent string. Promptwatch's AI crawler traffic report counts a request only when the source IP matches the operator's published ranges, so spoofed traffic is excluded. If you hand developers a spreadsheet of unverified bot hits, the first thing they will do is question it, and they will be right to.
Benchmark your logs against the market
The report covers weekly verified requests from June 8 to September 6, 2026. OpenAI made up 79.8% of verified AI crawler requests in the week of Aug 31 to Sep 6, down from 94.8% in Jun 8 to 14. The mix is shifting, so a one-time check goes stale.
The practical use is simple. Take the provider shares from the report and compare them with your own logs. If a provider is a large share in the market data but near zero for you, you are probably blocked or unreachable for that provider. Shares are a 100% mix, so a growing slice does not always mean more requests in absolute terms. Read the trend, not a single week.
Meta is a good example of why this changes quickly. Promptwatch Data on Meta-WebIndexer shows it going from about 2.2% of tracked AI crawler requests in mid-July 2026 to 37.8% on Aug 9, roughly 17x in under a month. At the peak it was the heaviest AI crawler in the logs. That is an indexing crawler only, not Meta's training or link-fetching bots.
This gives you two different asks for developers:
- Marketing ask: do not block these user agents in robots.txt, CDN or WAF rules if you want to be in Meta's index.
- Engineering ask: if your infrastructure bills per request, budget for sustained Meta crawl load.
Both are legitimate, and they pull in opposite directions. Surfacing the tradeoff with data beats a surprise bill or a silent block.
A prioritization framework developers will accept
Developers do not want a wishlist. They want a ranked list with a reason attached to each item. Here is the scoring I would use.
Step 1: Sort problems by type
Pull crawler logs for the last 30 days and bucket every issue:
- Access problems: 403, 401, 429 or challenge pages served to verified AI bots.
- Server problems: 5xx errors and timeouts on pages bots request.
- Discovery problems: pages never crawled because they are orphaned or missing from the sitemap.
- Content problems: pages crawled fine but rarely or never cited.
Access and server problems come first. They are binary, cheap to verify and usually small to fix. Content problems need editorial work and belong to marketing.
Step 2: Weight by business value
A crawl error on a pricing page matters more than one on a 2019 blog post. Tag each failing URL with a simple value: revenue page, lead page, support page, or low-value. Your analytics already know which is which.
Step 3: Weight by bot type
An error for a live-answer fetcher costs you an answer right now. An error for a training crawler is slower and less direct. Give user-triggered and search crawlers the heavier weight.
Step 4: Estimate effort with the developer
This is the part marketing cannot do alone. Show the ranked list and ask for rough sizing: an hour, a day, a sprint. A robots.txt change and a rendering rewrite should not sit on the same line.
| Fix type | Typical evidence in logs | Owner | Usual effort |
|---|---|---|---|
| robots.txt blocks a provider | Zero verified hits from a provider that is large in market data | Marketing plus dev | Hours |
| WAF or CDN rule challenges bots | 403 or 429 on verified AI IPs | Dev or infra | Hours to days |
| 5xx or timeouts on key pages | Server errors on pages bots request | Dev | Days |
| Pages missing from sitemap | Crawled rarely or never | Marketing plus dev | Hours |
| Client-side rendering hides content | Bot fetches page but content is absent from the response | Dev | Sprint |
| Thin or stale content | Crawled often, rarely cited | Marketing | Days |
Writing a ticket developers will not bounce
A good ticket has five lines:
- URL or URL pattern affected.
- Bot and provider, with the verified status code.
- Count and date range.
- Why it matters in business terms, one sentence.
- How you will confirm the fix.
For example: "ChatGPT-User received 403 on 214 requests to /pricing/* between Oct 1 and Oct 28. Likely a WAF rule. Pricing pages are our top converting page group. After the change we expect 200s in the crawler log within a day."
That reads like an incident, not a request. It also leaves developers to choose the solution, which they prefer.
Connect crawls to citations, not just to errors
A clean crawl does not guarantee a citation. This is where marketing teams stop too early. A page can be read ten times and never get cited, which points to content, not infrastructure.
Promptwatch Data on content half-life shows that pages ChatGPT cited on July 6, 2026 lasted very differently by type: landing pages and docs stayed cited longest, while news lost half its pages within 36 days. That tells you where refresh effort pays back and where it evaporates. And Promptwatch's content-type data found product pages were the most cited type in ChatGPT at 22.4% of citations from Sep 28 to Oct 4, 2026, so a broken product page template is rarely a minor bug.
The useful view is crawl to citation per page. Pages with many crawls and no citations go to the content team. Pages with no crawls go to the developers. Splitting the list this way ends a lot of arguments.
Tools for the job
You have a few categories to choose from, and most teams end up with a combination.
AI crawler and visibility platforms
Promptwatch is the one I would start with if you want crawler logs, citations and traffic in the same place. It records IP-verified AI crawler requests, explains crawl failures in plain language, and follows the crawl-to-citation path per page. It also ingests logs from Cloudflare, Vercel, Fastly and raw server logs, which means developers rarely need to build anything new. Its weekly site audit and Action Board then turn findings into a prioritized to-do list.
If you only want to see which bots visit, DarkVisitors is a lighter option focused on agents and bots.
Other AI visibility trackers such as Profound and Peec AI cover prompts and mentions well. Most stop at monitoring, so check whether they give you crawler-level evidence before you rely on them for dev tickets.
Traditional technical crawlers
These tools crawl your site as a bot would. They find broken links, redirect chains and orphaned pages, but they cannot tell you what AI bots really did on your server.
| Category | Answers | Misses |
|---|---|---|
| AI crawler log platform (Promptwatch, DarkVisitors) | Who visited, what they got, what failed | Depends on how logs are connected |
| Site crawler (Screaming Frog, Sitebulb, Lumar) | Site-level technical issues | What real AI bots experienced |
| Prompt tracker (Peec AI, Otterly.AI) | Whether you are mentioned | Why, and what to fix |
| Product analytics (Mixpanel, Amplitude) | On-site behavior after the click | How AI found you |
For wider browsing, the GEO software directory at bestgeosoftware.com lists more options by category.
Run the workflow monthly
A workable rhythm for a small team:
- Week 1: pull verified crawler data and compare provider mix with the market report. Flag any provider that is missing or erroring.
- Week 1: score failures by type, page value and bot type. Create tickets with the five-line format.
- Week 2: developers size and schedule. Marketing handles the content-side list in parallel.
- Week 3 to 4: confirm each fix in the logs. Did the status code change? Did crawls resume?
- Next month: check whether fixed pages started getting cited and sending visits.
Treat step four as non-negotiable. Developers like closing tickets with proof, and marketing gets a record that shows which fixes actually moved anything.
What this data will not tell you
Crawler data shows access, not influence. A bot reading your page does not mean the page is persuasive, and a citation does not mean a click. You still need citation tracking and visitor analytics to see the end of the chain, and the market mix itself moves week to week. Some of the most interesting claims, like Meta building its own index, rest on secondhand reports. Plan for the infrastructure effect, but do not rewrite strategy on a rumor.
A short checklist to start this week
- Confirm robots.txt, CDN and WAF rules for every provider in the list above.
- Pull 30 days of IP-verified crawler logs and group errors by URL pattern.
- Mark your ten most valuable pages and check each one for 200s from live-answer bots.
- Write three tickets in the five-line format and ask developers for sizing, not a promise.
- Book a check-in four weeks out to verify the logs and the citations.
Start with the 403s on your money pages. They are the cheapest wins you will find, and they will earn you the credibility to ask for the harder fixes later.