Key takeaways
- In Promptwatch Data,
meta-webindexerwent from roughly 2.2% of tracked AI crawler requests in mid-July 2026 to a peak of 37.8% on August 9, 2026. That is about 1 in 50 requests to more than 1 in 3. - Meta runs several crawlers with different jobs. Only Meta-WebIndexer is documented as the one that helps Meta AI cite and link to your content.
- Meta-ExternalFetcher and FacebookExternalHit may bypass robots.txt, so a Disallow rule will not fully silence everything with "meta" or "facebook" in the user agent.
- Some site owners report extreme load, including 7.9 million requests in 30 days and 230,743 requests in a day and a half. Check bandwidth before it surprises you.
- User agent strings can be faked. Verify against Meta's published IP ranges before you trust a count.
What happened with Meta-WebIndexer
In mid-July 2026, Meta-WebIndexer was a minor player. According to Promptwatch crawler data, it made up about 2.2% of AI crawler requests in the logs the platform tracks. The share sat around 2% until July 17, spiked to roughly 23% between July 20 and 22, then climbed again from August 5 and hit 37.8% on August 9.
That makes it the heaviest AI crawler in those logs at the peak. You can see the daily chart in Promptwatch's Meta-WebIndexer report.

Why is it happening? Nobody outside Meta knows for sure. On August 6, the developer @levelsio posted that Meta staff told him Meta is "allegedly" building its own web index, and that Meta's crawling was heavy enough to trigger load alerts on his VPS. That is unconfirmed. What Meta does document is the stated purpose of the crawler: improving Meta AI search result quality.
I would treat the surge as a signal, not a verdict. Crawling at this pace looks like index building, but the only thing you can act on is what lands in your own logs.
Meta's crawlers and what each one does
The most common mistake is lumping every Meta user agent together. Meta's developer documentation (updated May 21, 2026) separates them by purpose.

| Crawler | User agent token | Documented purpose | Respects robots.txt? |
|---|---|---|---|
| Meta-WebIndexer | meta-webindexer/1.1 | Improve Meta AI search quality; allowing it helps Meta cite and link to your content | Yes |
| Meta-ExternalAgent | meta-externalagent/1.1 | Train foundation AI models, or improve products by indexing content | Yes |
| Meta-ExternalAds | meta-externalads/1.1 | Improve advertising and business products | Yes |
| Meta-ExternalFetcher | meta-externalfetcher/1.1 | Fetch individual links at a user's request, supports agentic features | May bypass |
| FacebookExternalHit | facebookexternalhit/1.1 | Link previews for shares on Facebook, Instagram, Messenger | May bypass for security checks |
Promptwatch's share figure counts only meta-webindexer in the numerator. The denominator is every AI crawler it identifies by user agent, including OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot, PerplexityBot, Google-Extended and others. Meta's other bots (meta-externalagent, meta-externalfetcher, FacebookBot) sit in the denominator only. Googlebot, Bingbot, FacebookExternalHit and humans are excluded.
So if your logs show a different percentage, that is expected. The 38% is a share of AI crawler requests in one data set, not of your total traffic.
How to find Meta-WebIndexer in your server logs
Start with the raw access logs. You want the user agent field, which in the common combined log format is the last quoted string.
Count requests by Meta user agent
# Requests per Meta user agent (nginx/Apache combined format)
grep -iE 'meta-(webindexer|externalagent|externalfetcher|externalads)|facebookbot|facebookexternalhit' access.log \
| awk -F'"' '{print $6}' \
| sed -E 's/\/[0-9.]+.*//' \
| sort | uniq -c | sort -rn
This gives you a ranked list of which Meta bot is doing the work. If meta-webindexer dominates, you are seeing the same pattern Promptwatch reports.
Chart it by day
grep -i 'meta-webindexer' access.log \
| awk '{print $4}' | cut -d: -f1 | tr -d '[' \
| sort | uniq -c
Look for step changes. In the aggregate data the share jumped around July 20 to 22 and again from August 5. Your own site may show different timing, or a single burst.
See what it is requesting
grep -i 'meta-webindexer' access.log \
| awk '{print $7}' | sort | uniq -c | sort -rn | head -50
This is where trouble shows up. Faceted navigation, calendar pages, search result URLs, old revisions and diff pages can eat a crawler's attention and your CPU.
Check whether it ever reads robots.txt
grep -i 'meta-webindexer' access.log | grep -c 'robots.txt'
This check matters. One site owner writing on Towards AI reported that meta-webindexer made 230,743 requests in about a day and a half, 98% of them (226,391) to paths disallowed in robots.txt, at hundreds per minute despite a ten second crawl-delay request, and that none of those requests was for robots.txt itself. That is one operator's account from their own logs and AWS bills, not proof of how Meta behaves everywhere. But it tells you what to test on your own site rather than assume.
A Reddit post on r/webdev described a different pattern: 7.9 million requests in 30 days and more than 900 GB of bandwidth before the owner noticed. Again, single-site anecdotes. They are useful because they show the cost side: if your host bills by request or egress, a crawler at this volume is a budget line.
Verify it is really Meta
A user agent is a claim. Anyone can send meta-webindexer/1.1. Before you block or rate-limit on a count, confirm the source.
- Pull the source IPs for the requests you care about.
- Compare them with the IP ranges Meta publishes for its crawlers. Meta's docs point to an identify section for this and recommend IP allow-listing as more secure than user agent matching. I could not confirm in the scraped text exactly how Meta publishes those ranges, so check the docs directly.
- Run a reverse DNS lookup on a sample of IPs as a sanity check.
- Treat unverified hits as spoofed or unknown traffic, and count them separately.
This is also why two reports can disagree. Promptwatch's Meta-WebIndexer report identifies crawlers by user agent. Its separate AI crawler traffic report (June 8 to September 6, 2026) counts only IP-verified requests from OpenAI, Anthropic, Google, Perplexity and Mistral, and does not list Meta. The denominators differ, so do not stack the two sets of percentages on top of each other.
Allow, limit or block: a decision framework
There is no universal answer. It depends on what you want from Meta AI and what the crawl costs you.
Reasons to allow it
Meta says allowing Meta-WebIndexer "helps us cite and link to your content in Meta AI's responses." If Meta AI becomes a real discovery surface, being crawled now is a precondition for being cited later. If you already care about being cited in ChatGPT and Perplexity, the logic is the same.
Reasons to limit or block it
- Your infrastructure bills per request or per GB, and the crawl is expensive.
- The crawler is hammering low-value URLs like faceted filters, internal search or revision history.
- You do not want content used for AI products, in which case you will also want to decide about Meta-ExternalAgent, which is documented for training and product indexing.
Robots.txt examples
Allow the indexer, block training:
User-agent: meta-webindexer
Allow: /
User-agent: meta-externalagent
Disallow: /
Allow the indexer but keep it out of expensive paths:
User-agent: meta-webindexer
Disallow: /search
Disallow: /*?filter=
Disallow: /index.php
Crawl-delay: 10
Crawl-delay is a non-binding convention and plenty of crawlers ignore it, so do not rely on it as protection. Meta says crawlers may cache robots.txt for up to 24 hours, so changes are not instant. If you have a specific problem, Meta lists [email protected] as a contact.
When robots.txt is not enough
Meta-ExternalFetcher and FacebookExternalHit may bypass robots.txt. FacebookExternalHit fetches pages when someone shares a link, so blocking it breaks link previews on Facebook, Instagram and Messenger. If you need hard limits, use your CDN or WAF: rate limit by verified IP range, return 429 or 403 to the specific user agent, and cache aggressively on the paths it favors.
You can test how your server responds to the preview crawler with Meta's own command:
curl -v --compressed -H "Range: bytes=0-524288" -H "Connection: close" \
-A "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" "$URL"
Meta's docs also note your server should support gzip and deflate and keep Open Graph tags within the first 1 MB.
Compare your logs with the wider picture
Promptwatch's crawler report gives a useful rule of thumb: if a provider is large in the benchmark but near zero in your logs, you are probably blocked or unreachable for it. Check robots.txt, CDN and WAF rules. The same idea applies in reverse. If Meta-WebIndexer is huge for you but you never meant to allow it, find out which rule let it in.
For context, the same IP-verified report shows OpenAI at 79.8% of verified requests in the week of August 31 to September 6, down from 94.8% in the week of June 8 to 14. The mix is moving, and Meta is one reason to expect it to keep moving. Promptwatch also tracked a related shift in how ChatGPT searches, covered in its site: operator report, which is a reminder that crawler and citation behavior can change overnight.
Tools for reading crawler traffic
Raw logs work for a one-off check. For ongoing monitoring you want something that classifies bots, verifies IPs and ties crawling to citations.
| Tool | What it does for crawler analysis | Good fit |
|---|---|---|
| Promptwatch | IP-verified AI crawler logs, crawl failures explained, crawl-to-citation path per page | Teams that want to connect crawling to AI citations and traffic |
| DarkVisitors | Tracks AI agents, bots and LLM referrals visiting your site | Quick bot visibility |
| Screaming Frog | Log file analysis alongside site crawling | Technical SEOs comfortable with desktop tooling |
| Oncrawl | Log analysis for large sites | Enterprise sites with heavy crawl budgets |
| Botify | Enterprise log analysis with AI search focus | Very large sites |
Promptwatch logs GPTBot, ClaudeBot, PerplexityBot, Meta-WebIndexer, Meta-ExternalAgent and other crawlers, verifies visits against each operator's published IP ranges so spoofed bots drop out, and shows which pages were read and where crawls failed. Setup takes minutes through your CDN, host or the WordPress plugin.

If you want a lighter option for bot tracking:

For classic log analysis in a crawler you may already own:

For enterprise-scale log work:
If you are comparing options more broadly, the GEO software directory lists more platforms.
A practical checklist
- Run the grep commands above over the last 30 days and note requests, bandwidth and top paths per Meta user agent.
- Verify a sample of IPs against Meta's published ranges.
- Check whether
meta-webindexerever fetches robots.txt on your site. - Decide your policy per bot: indexer, training, fetcher, previews. They are separate decisions.
- Write robots.txt rules, then add WAF or CDN rate limits for anything that needs a hard stop.
- Wait at least 24 hours, then recheck the logs.
- Set an alert on requests per user agent so the next surge does not wait a month to be noticed.
- If you allow the indexer, watch for Meta AI citations over the following weeks. Crawling is necessary, not a guarantee.
The summary: Meta-WebIndexer is now a large share of AI crawler traffic in at least one major data set, it is documented as feeding Meta AI citations, and the only way to know what it costs you is to read your own logs.

