Key takeaways
- Meta documents five crawlers: FacebookExternalHit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher. They do different jobs, so a blanket block is a blunt tool.
- Meta-WebIndexer is the visibility crawler. Meta says it improves Meta AI search quality and that allowing it helps Meta cite and link to your content.
- Meta-ExternalAgent is the training and indexing crawler. Seeing it in your logs tells you your content may be used for foundation models, not that you will be cited.
- Per Promptwatch Data, Meta-WebIndexer went from about 2.2% to 37.8% of tracked AI crawler requests between mid-July and August 9, 2026.
- Crawl frequency is not citation. Logs show access. Only prompt tracking shows whether Meta AI actually names you.
Why Meta's crawlers deserve a look in your logs
Most AI visibility conversations revolve around ChatGPT, Perplexity and Google. Meta AI gets less attention, even though Meta clearly wants it to be a search-capable assistant. Your server logs are the earliest signal of how seriously Meta is treating your site, and they cost nothing to read.
The tricky part is that Meta does not use one crawler. It uses several, and the user agent tells you which job a request is doing. Treating them all as "Meta scraping me" leads to bad decisions, like blocking the bot that generates link previews or the one that might get you cited.
The page above is Meta's own reference and the primary source for everything in the next section. It is worth bookmarking, since Meta updates it (the version I read was updated May 21, 2026).
The five Meta crawlers and what each one means
| Crawler | User agent token | Documented purpose | What it signals for you |
|---|---|---|---|
| FacebookExternalHit | facebookexternalhit/1.1 | Builds title, description and thumbnail when a link is shared on Facebook, Instagram or Messenger | Someone shared your URL. Not an AI signal. |
| Meta-WebIndexer | meta-webindexer/1.1 | Improves Meta AI search result quality; helps Meta cite and link content | You are being considered as a Meta AI source |
| Meta-ExternalAgent | meta-externalagent/1.1 | Training foundation AI models or indexing content directly | Your content may feed model training and product indexing |
| Meta-ExternalAds | meta-externalads/1.1 | Improving advertising and business products | Ads and business product crawling |
| Meta-ExternalFetcher | meta-externalfetcher/1.1 | Fetches individual links at a user's request, including agentic tasks | A user or agent asked for your page. May bypass robots.txt. |
A few details matter more than the table can show.
FacebookExternalHit may also bypass robots.txt for security or integrity checks, such as malware detection. Meta-ExternalFetcher may bypass robots.txt because the fetch was requested by a user. So robots.txt is your main control for Meta-WebIndexer and Meta-ExternalAgent, but not a guarantee for the other two.
Meta also says it prefers industry-standard robots.txt over non-standard formats like NoAI tags. Don't bother with NoAI meta tags for Meta.
Reading Meta-WebIndexer in your logs
This is the one I would watch most closely. Meta describes it as the crawler that navigates the web "to improve Meta AI search result quality", and says that allowing it "helps us cite and link to your content in Meta AI's responses."
In your access logs it looks like this:
meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers)
meta-webindexer/1.1
Some third-party sources list a full URL in the parenthetical instead. Match on the meta-webindexer token rather than the whole string.
What Promptwatch's data shows
Promptwatch Data tracked a sharp change in this crawler's behavior. In mid-July 2026, Meta-WebIndexer was about 2.2% of AI crawler requests in Promptwatch's logs. It hovered near 2% until July 17, spiked to roughly 23% between July 20 and 22, passed 26% around August 6, and peaked at 37.8% on August 9. That is about 1 in 50 requests becoming more than 1 in 3, and it made Meta-WebIndexer the heaviest AI crawler in the dataset. You can see the daily chart in Promptwatch's Meta-WebIndexer report.
Two caveats I would keep in mind:
- The percentage is Meta-WebIndexer requests divided by all tracked AI crawler requests. It is not a share of total site traffic, and Googlebot, Bingbot and FacebookExternalHit are excluded.
- That report matches on user agent. Promptwatch's separate AI crawler traffic report uses IP-verified requests, and Meta is not among the providers charted there. So some Meta numbers could include spoofed requests. I could not find published Meta IP ranges, which makes verification harder than with other operators.
Is Meta building a search index?
On August 6, 2026, @levelsio posted that Meta staff had allegedly told him the company is building its own web index so its AI doesn't have to search through Google. He also said Meta was crawling his sites hard enough to trigger load alerts. Promptwatch says the surge "lines up with" such reports.
Treat this as unconfirmed. I found no Meta statement confirming a search index. But the behavior is consistent with the documented purpose: a crawler that exists to improve Meta AI search, suddenly fetching a lot of pages.
The practical reading is simple. If Meta is assembling an index now, pages it can't fetch now may be absent when Meta AI search gets heavier use.
Reading Meta-ExternalAgent in your logs
Meta-ExternalAgent is documented as crawling "for use cases such as training foundation AI models or improving products by indexing content directly." The string looks like:
meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers)
meta-externalagent/1.1
The common 51Degrees-observed variant carries a developers.facebook.com URL in the parenthetical, so again, match the token.
What does it tell you about visibility? Less than people hope. Meta ties Meta-WebIndexer to citations. It does not make that promise for Meta-ExternalAgent. Heavy Meta-ExternalAgent traffic means your content is being collected for training and indexing. It does not mean Meta AI will name your brand.
The volume problem
This crawler gets complaints. A Reddit thread on r/webdev described Meta's AI crawler hitting a site 7.9 million times in 30 days, and a Cloudflare community post describes repeated traffic from meta-externalagent and meta-externalfetcher. I only have snippets of both, so I can't vouch for the details. But if you bill per request, or run a small server, the load is a real cost.
Single Grain's write-up also notes that Meta's new commercial terms include a crawling clause, with an effective date that has shifted between Oct 30 and Nov 20, 2026. I could not read the full terms text, so check Meta's terms page yourself before making legal decisions.
Meta-ExternalFetcher and what a user-triggered fetch means
Meta-ExternalFetcher is the odd one. It fires when a user asks for a specific link, and Meta says it supports evaluating and improving agentic AI capabilities, including helping AI navigate sites to complete tasks.
For visibility, a hit from this crawler is arguably the most interesting. Somebody, or some agent, asked for your page. That is closer to demand than to bulk collection. It is also the one you cannot reliably block with robots.txt.
What to look for in your logs
You don't need a fancy setup. A few steps get you most of the way.
- Filter logs for the five tokens: facebookexternalhit, meta-webindexer, meta-externalagent, meta-externalads and meta-externalfetcher. Promptwatch also lists FacebookBot in its tracked Meta set.
- Count requests per token per week. The ratio between them matters more than any single number.
- Check status codes. A wall of 403 or 429 responses to meta-webindexer means your WAF is blocking the one crawler tied to citations.
- Look at which paths get fetched. Are they your commercial pages, docs and product pages, or only a parameter-heavy corner of the site?
- Compare against the mix of other AI crawlers. Promptwatch's advice applies: if a provider has a big share in industry data but near zero in your logs, you are probably blocked or unreachable for it.
To simulate what Meta's link-preview crawler sees, Meta's own docs give a curl command:
curl -v --compressed -H "Range: bytes=0-524288" -H "Connection: close" -A "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" "$URL"
For FacebookExternalHit specifically, Meta also asks that your server supports gzip and deflate, that Open Graph tags appear in the first 1 MB, and that the page responds within a few seconds.
Tools that help
If you'd rather not grep raw logs, a few options exist. Promptwatch includes IP-verified AI crawler logs that show which pages AI bots read, which errors they hit, and a crawl-to-citation path per page, with spoofed bots filtered out.
DarkVisitors is a lighter option focused on tracking AI agents and bots visiting a site.
For crawl-side technical diagnosis, a log file analyzer such as JetOctopus or Botify can segment bot traffic at scale.
Setting a policy that matches your goals
There is no universal answer, but the logic is straightforward once the crawlers are separated.
| Your priority | Meta-WebIndexer | Meta-ExternalAgent | FacebookExternalHit |
|---|---|---|---|
| Maximum Meta AI reach | Allow | Allow | Allow |
| Reach, but no training use | Allow | Block | Allow |
| Content is the product (research, paid data, courses) | Decide case by case | Block | Allow |
| Server load is the problem | Rate limit, don't block | Block or rate limit | Allow |
My own lean for most brands is the second row. Allow the crawler Meta ties to citations, block the training crawler if that bothers you, and leave link previews alone.
A robots.txt for that looks like this:
User-agent: meta-webindexer
Allow: /
User-agent: meta-externalagent
Disallow: /
User-agent: facebookexternalhit
Allow: /
There is a catch. Meta's documentation says Meta-ExternalAgent is used for training and for "improving products by indexing content directly." Blocking it may therefore have effects on product indexing that Meta has not spelled out. I can't tell you precisely how that interacts with Meta AI citations, and Meta's docs don't either. If Meta AI visibility is a priority, test your changes and watch results rather than assuming.
Also remember that robots.txt is a request, not a wall. Your CDN or WAF rules are what actually stop traffic, and they can silently block crawlers you meant to allow.
Crawled is not the same as cited
This is the part people skip. A crawler hit proves Meta fetched your page. It says nothing about whether Meta AI uses it in an answer.
The pattern holds for other AI systems too. Promptwatch's data on ChatGPT citation share by content type shows that product pages (22.4% of citations, Sep 28 to Oct 4, 2026), landing pages and listicles earn most citations, and not every crawled page does. Content that gets fetched still has to be useful, specific and structured to win a slot. Meta AI will likely have its own preferences, which nobody has measured publicly yet.
So pair your log work with output tracking. Run the prompts your buyers actually ask in Meta AI and see whether you show up. Platforms that track Meta AI through its real interface, like Promptwatch, can do this at scale and connect crawl activity to citations. Other trackers in the space, such as Profound or Peec AI, vary in which assistants they cover, so check Meta AI support before you buy.
The GEO software directory at bestgeosoftware.com is a decent place to compare platforms by which AI engines they actually monitor.
A simple monitoring routine
- Weekly: count requests per Meta token and flag any status code spikes.
- Monthly: compare the Meta-WebIndexer share of your AI bot traffic against the broader picture in Promptwatch's crawler reports.
- After any robots.txt, CDN or WAF change: re-check logs within a few days to confirm the right bots still get through.
- Quarterly: re-read Meta's crawler documentation page, since user agents and descriptions change.
What I would do this week
Pull seven days of logs. Split Meta traffic by token. If meta-webindexer is returning 403s, fix that first, because it is the only Meta crawler Meta explicitly links to citations. Then decide on meta-externalagent based on whether training use bothers you or the request volume bothers your hosting bill. Leave facebookexternalhit alone unless you have a very good reason.
The surge in Meta-WebIndexer traffic might be the start of a real Meta AI search index, or it might be a one-off crawl. Nobody outside Meta knows yet. What you do control is whether your pages are reachable when it matters.