Key takeaways
- A crawl failure explanation is a diagnosis of which layer stopped the bot: robots.txt, CDN or WAF, rate limiting, the HTTP response, or the page itself.
- A 403 or 429 is an access decision. A 200 with an empty body is a rendering problem. Mixing them up wastes the most time.
- robots.txt is almost never the cause of a 403. Start at the CDN and WAF.
- Most AI crawlers read raw HTML and do not run JavaScript, so a page that looks fine to Googlebot can look empty to them.
- Test from outside your network, then confirm in your logs that bots now get 200s.
What a crawl failure explanation actually tells you
A crawl failure explanation is the plain-language reason a bot could not read a page. It usually combines three things: the status code the bot received, the layer that produced it, and what the bot saw in the body (if anything). Your tool might phrase it as "blocked by firewall", "rate limited" or "page returned no content". Those are three different problems with three different owners.
The habit worth building is to read the explanation as a pointer to a layer, not as a verdict on your content. Most AI visibility problems are access problems. Your pages can be excellent and still never get cited, because the bot never got the page.
If you want data on how much is at stake, Promptwatch Data's AI crawler traffic report (June 8 to September 6, 2026) counted only requests from IPs that match each operator's published ranges. OpenAI made up 79.8% of verified AI crawler requests in the week of Aug 31 to Sep 6, down from 94.8% in early June. The report's practical advice is simple: compare that mix with your own logs. If a provider is a big share in the report and near zero in your logs, you are probably blocked or unreachable for it.
The five layers a bot request has to pass
Every AI crawler request goes through the same gauntlet. A block at any point stops it, and only the first is visible in robots.txt.
| Layer | What it controls | Typical symptom | Who usually owns it |
|---|---|---|---|
| robots.txt | Polite opt-in/opt-out by user agent | Bot never requests the page | SEO or web team |
| CDN / WAF | Bot toggles, managed rules, reputation lists | 403 or a challenge page | Infrastructure or security |
| Rate limits | Requests per IP or ASN | 429 or 503 | Infrastructure |
| HTTP response | Status code, redirects, errors | 404, 5xx, redirect loops | Engineering |
| Page rules | noindex, nosnippet, X-Robots-Tag, rendering | 200 but empty or excluded | Content or engineering |
Read the table left to right when you troubleshoot. The layer most often missed is the CDN/WAF, because a robots.txt test cannot see it. Only a live fetch as the bot can.
Decode the status code first
Before anything else, get the status code the bot received. It narrows the search faster than any other single fact.
403: a decision, not a bug
A 403 is made before your page runs. It comes from a firewall rule, a bot-management feature, a rate limiter or a security plugin. It does not come from robots.txt, which is why checking that file first is a classic time sink. People check it because they know how to read it.
Two patterns help:
- If every AI crawler is blocked, look for a broad bot-management setting at your CDN.
- If only one is blocked, look for a rule naming that bot, or a reputation list that contains its addresses.
Defaults matter here. Cloudflare's own 2026 post on AI traffic options describes the shift from a one-click "block AI bots" posture toward classifying bots by what they do on your site: what they store, and how they reshare content. In practice that means a setting someone enabled for good reasons (keeping training bots off) can also catch bots you want, such as answer-time fetchers. Check whether you inherited a default rather than wrote a rule.
429 and 503: you are rate limiting them
These mean the server or CDN is throttling the bot per IP or per network. AI crawlers often come from a small set of address ranges, so a limit tuned for human traffic can trip quickly. Look at the limit threshold and whether the crawler's published ranges can be exempted.
404 and 410: the bot asked for something that is not there
Bots frequently request URLs that do not exist, from stale links, old sitemaps or guessed paths. JetOctopus's 2026 guide on AI bots suggests listing the 404 URLs a bot requests and checking response times by section, using about 0.8 seconds as a rough yardstick. Slow templates are worth fixing because timeouts look like failures from the bot's side.
3xx: usually fine, until it is a chain
A single redirect is normal. Chains and loops waste the bot's patience. Point links and sitemaps at final URLs.
200 with nothing useful
This is the sneaky one. The status says success, yet the body is an empty shell or a challenge page. That is a rendering or bot-challenge problem, not an access code problem. Crawlbase's engineering write-up makes the same point from the scraping side: separate the fetch verdict from the target's answer, and check whether the data you expect is actually present.
A diagnostic order that narrows fast
The sequence below is ordered by how much each step rules out, not by how likely each cause is.
- Reproduce the failure as the bot experiences it. Request the URL with the crawler's user-agent string and without JavaScript, and record the status code and body. If you cannot reproduce it deterministically, you cannot confirm it is fixed.
- Separate status-code problems from content problems. A 403 or 429 is an access decision. A 200 with an empty shell is rendering.
- Check the layers in front of your application: CDN rules, WAF policies, rate limits, bot-management features. These are often configured by a different team.
- Rule robots.txt in or out explicitly. Test the exact failing path against the live file. If it permits the path, stop looking there.
- Verify from outside your network. Testing from inside the perimeter, or from an already allowlisted address, reports success that external crawlers do not experience.
Here is a quick way to reproduce step 1 from a terminal:
curl -sS -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)" \
https://example.com/your-page
Run it from a network that is not on your allowlist, and compare the result with the same request using a normal browser user agent. A difference between the two is your clue. Be aware that a spoofed user agent from an unknown IP can be treated differently from the real bot, since many WAFs verify source IPs against published ranges. A matching result is informative, a mismatch means you also need log evidence from the real crawler.
When the problem is rendering, not access
JetOctopus's measurement summary says it plainly: most AI crawlers read raw HTML and did not run JavaScript. If your product copy, pricing or article body is injected client-side, a page that looks complete to Googlebot can look empty to GPTBot or PerplexityBot.
Signs you have this problem:
- Bot logs show 200 responses, but the page never gets cited.
- View-source shows a near-empty div and a bundle of scripts.
- A JavaScript-off fetch returns far less text than the rendered page.
Fixes include server-side rendering, static generation, or pre-rendering the critical content. A crawler like JetOctopus can compare raw and rendered HTML across a site, and its log analyzer shows what bots actually received.
Read your logs, not just your config
Config tells you what you intended. Logs tell you what happened. For AI crawlers, two details matter:
- Verify the source IP against the operator's published ranges. Plenty of requests claim to be GPTBot and are not. Promptwatch's crawler data filters spoofed bots for this reason.
- Look at which bots, not just which provider. Provider totals hide the split between training crawlers and answer-time fetchers, and you may want to allow one and not the other.
A concrete example of why monitoring matters: Promptwatch Data's Claude citation crawler report shows Claude-User going from about 30 visits a day in mid-December 2025 to several thousand a day by mid-April 2026, more than 100x growth. It is still around 1% of citation crawler traffic, but a bot that grows that fast is one you want to confirm is not blocked. The report recommends checking that Claude-User and ClaudeBot are not blocked by robots.txt or CDN/WAF bot rules, and baselining Anthropic user agents in your logs, reviewing monthly.
Promptwatch ties crawler logs to citations, so you can see the path from a bot reading a page to that page being cited, and spot pages that are crawled but never cited. If you are only getting a vague "blocked" signal from another tool, that crawl-to-citation view is where the explanation gets specific.
Other tools worth knowing for bot traffic visibility include DarkVisitors, which tracks AI agents and bots visiting a site.
Common crawl failure explanations and what to do
| What the explanation says | Likely layer | First thing to check | Typical fix |
|---|---|---|---|
| Blocked by robots.txt | robots.txt | The exact User-agent group and path | Add an Allow or fix the wildcard group |
| 403 Forbidden | CDN / WAF | AI bot toggles, managed rules, security plugin | Allow the specific bot or its verified IPs |
| 429 Too many requests | Rate limit | Per-IP and per-ASN thresholds | Raise or exempt verified crawler ranges |
| 5xx or timeout | Origin or HTTP | Slow templates, origin capacity | Fix slow pages, cache, scale |
| 404 on bot requests | HTTP | Stale sitemap and old links | Redirect, remove from sitemap |
| Challenge page returned | CDN / WAF | Bot-fight or JS challenge on those paths | Skip the challenge for verified bots |
| 200 but empty content | Rendering | Raw HTML vs rendered HTML | Server-render critical content |
| noindex or nosnippet found | Page rules | Meta tags and X-Robots-Tag headers | Remove directives that were added by mistake |
Fix it without opening the door to everything
Unblocking a bot is not the same as unblocking every bot. A few habits keep you safe:
- Allow by verified identity, not by user agent alone. User-agent strings are trivial to fake. Pair them with IP range checks.
- Decide per bot purpose. Training crawlers, search indexers and user-triggered fetchers are different. You might welcome the last two and still restrict the first.
- Keep robots.txt as the polite layer, and use the CDN to enforce. Major operators such as OpenAI, Anthropic, Google and Perplexity state that their crawlers honor robots.txt, but a bot that does not honor it will only be stopped at the edge.
- Exempt, do not disable. Add narrow exceptions rather than switching off bot protection.
Confirm the fix
The last step is the one people skip. After each change:
- Re-run the external request and record the new status code.
- Check that the bot's real requests in your logs now return 200s. JetOctopus's guide recommends repeating after the next crawl and confirming that bots now receive 200 responses.
- Turn the biggest problems into backlog tickets with owners. A ranked list by pages affected, or by Search Console impressions, helps you pick the top three.
- Watch for regressions after releases, since CDN rule changes tend to come back.
For technical crawling and monitoring, tools like Screaming Frog and Sitebulb help with the raw-versus-rendered side, and ContentKing watches for changes in real time.
If you want more options in the category, the GEO software directory at bestgeosoftware.com lists tools that track AI visibility and crawler activity.
A short checklist to keep
- Get the status code the bot received, from the bot's point of view.
- Decide: access (403, 429, 5xx) or content (200, empty)?
- Work outward from the CDN and WAF. Leave robots.txt alone until those are cleared, unless the explanation names it.
- Verify with an external request and with real log entries from verified IPs.
- Review monthly. New bots show up, and defaults change.
A crawl failure explanation is only useful if it sends you to the right layer. Read the code, name the layer, find its owner, and fix that one thing. Then check the logs.