Key takeaways
- Meta runs several crawlers with different jobs. Meta-WebIndexer is the one tied to Meta AI search results and citations. Meta-ExternalAgent is for training, and Meta-ExternalFetcher fetches pages on a user's request.
- Meta says allowing Meta-WebIndexer in robots.txt "helps us cite and link to your content in Meta AI's responses." Blocking it is the fastest way to make yourself invisible there.
- Per Promptwatch Data, Meta-WebIndexer reached 37.8% of tracked AI crawler requests on August 9, 2026, up from about 2.2% in mid-July.
- Meta has not published citation ranking signals. Everything beyond crawler access is general AI-crawler hygiene: server-rendered HTML, clean structure, fast responses, direct answers.
- Check your own server logs. If Meta bots hammer your site, you may need caching and rate limits.
Why Meta AI deserves a spot on your list
Most GEO advice is about ChatGPT, Perplexity and Google. Meta AI gets less attention, and that is exactly why a quick look at your crawler settings is worth an afternoon.
The trigger is a crawler surge. Promptwatch Data's Meta-WebIndexer report shows the meta-webindexer user agent hovering around 2% of tracked AI crawler requests until July 17, 2026, spiking to about 23% between July 20 and 22, then climbing to 37.8% on August 9. That is roughly 17x in under a month, and it made Meta-WebIndexer the heaviest AI crawler in the logs. The report links the surge to reports that Meta is building its own web search index so Meta AI does not depend on Google or Bing.
A caveat I want to be upfront about: that explanation is unconfirmed. On August 6, 2026, @levelsio posted that Meta was scraping his sites hard enough to trigger server load alerts, and that Meta staff told him Meta is "allegedly" building its own index. It is one anecdote. I also found no Meta documentation on how Meta AI ranks or picks citations, and no published data on Meta AI citation rates. So this guide is honest about what is known (crawler access) and what is reasonable inference (content practices that help every AI system).
Meet Meta's crawlers
Meta documents its crawlers on its web crawlers page for webmasters. The four you will see in logs:
| Crawler | User agent | Purpose | Respects robots.txt? |
|---|---|---|---|
| Meta-WebIndexer | meta-webindexer/1.1 | Crawls the web to improve Meta AI search result quality | Not on Meta's bypass list; no independent audit confirms compliance |
| Meta-ExternalAgent | meta-externalagent/1.1 | Training foundation AI models, or indexing content directly | Listed as a robots.txt-governed crawler |
| Meta-ExternalFetcher | meta-externalfetcher/1.1 | Fetches individual links at a user's request, supports agentic tasks | May bypass robots.txt, per Meta |
| FacebookExternalHit | facebookexternalhit | Link previews and checks | May bypass robots.txt for security or integrity checks |
Two things stand out. First, only Meta-WebIndexer is explicitly described as the thing that helps Meta AI cite and link to you. Second, the fetchers can ignore robots.txt by Meta's own admission, so robots.txt is not a complete control.
A few more notes from secondary sources, which I treat as context rather than fact. The ai.robots.txt registry added Meta-WebIndexer on September 8, 2025 with compliance marked "unclear". Drop Site News reported in August 2025, via leaked documents, that Meta harvested content from 6 million unique websites and that its scraping may have ignored robots.txt. That is a reported claim, not a proven one.
Step 1: Set your robots.txt deliberately
The decision is a policy choice, not a technical one. Decide what you want from Meta:
- Want to appear in Meta AI answers with a link? Allow
meta-webindexer. - Do not want your content used for training? Disallow
meta-externalagent. - Want link previews on Facebook and Instagram? Keep
facebookexternalhitallowed.
A setup that does all three, adapted from an example by SingleGrain:
User-agent: meta-externalagent
Disallow: /
User-agent: meta-webindexer
Allow: /
User-agent: facebookexternalhit
Allow: /
There is a wrinkle. Meta describes Meta-ExternalAgent as crawling for training "or for improving products by indexing content directly." So blocking it may have effects beyond training. That is my inference, not a Meta statement, but it is worth testing before you assume a clean split.
Other practical points:
- Meta prefers robots.txt over non-standard formats such as "NoAI tags." Do not rely on those.
- Crawlers cache robots.txt for up to 24 hours, so allow a day before judging a change.
- The product token is case-insensitive under RFC 9309, so
Meta-WebIndexerandmeta-webindexermatch the same rule. - Check that a blanket
User-agent: *Disallow, or a CDN or WAF bot rule, is not silently blocking Meta anyway. This is the most common reason a site is invisible to a crawler it thinks it allowed.
If you want to generate rules for many bots at once, 51Degrees offers a free robots.txt generator covering 429 crawlers, and Promptwatch has a free Robots.txt Generator for AI.
Step 2: Read your logs
Before optimizing anything, find out whether Meta is already visiting. Search server or CDN logs for meta-webindexer, meta-externalagent, meta-externalfetcher and FacebookBot. Look at three things: how many requests, which URLs, and what status codes come back.
I could not verify Meta's published IP ranges, so I cannot give you a verified-IP recipe for Meta. User agents are easy to spoof, which means your raw log counts include some fakes. Treat them as directional.
For context on the wider mix, Promptwatch's AI crawler traffic report covers June 8 to September 6, 2026 and counts only requests whose source IP matches the operator's published ranges. OpenAI made up 79.8% of verified requests in Aug 31 to Sep 6, down from 94.8% in Jun 8 to 14. Meta is not in that verified-IP view, so the Meta-WebIndexer numbers come from the user-agent-based report above. The practical use of the verified report: if a provider is large there but near zero in your logs, you are probably blocked or unreachable. Check robots.txt, CDN and WAF rules.
Tools that help with log analysis:
Promptwatch has AI crawler logs that work with Cloudflare, Vercel, Fastly and raw server logs, and show which pages bots read and which errors they hit.
DarkVisitors is a lighter option for tracking AI agents and bots on your site.
If you want a classic crawler to see your site the way a bot would, Screaming Frog and Sitebulb are the usual choices.
Step 3: Watch the crawl load
A surge like this has a cost. The Promptwatch report recommends budgeting for sustained crawl load if your infrastructure bills by request. A thread on r/CloudFlare describes sites with very heavy Meta crawling, including one claim of 7.9 million requests in 30 days and 900+ GB. I did not open the thread and those are anecdotal user claims, but the pitfall is real enough to plan for.
What to do about it:
- Cache aggressively at the CDN so repeated bot hits do not reach your origin.
- Rate limit by user agent rather than blocking outright, if you want citations but not load.
- Fix 404 sources. AI crawlers fetch more 404s than traditional search bots, often requesting outdated assets, so keep redirects in place for changed URLs.
- Keep sitemaps current and URL patterns consistent so crawlers stop guessing.
Step 4: Serve content in the HTML
Here I am leaning on general AI-crawler guidance, not Meta documentation. I found nothing saying whether Meta's crawlers render JavaScript. The safe assumption, the one most AI crawler guides make, is that they do not.
A Search Engine Land guide on AI crawler optimization states that, unlike Google, AI crawlers rely on the HTML response. If critical content or links only appear after JavaScript rendering, most AI crawlers cannot access them. Another guide in our research cites 57.7% of ChatGPT's fetches being HTML content.
Test it quickly:
- View source (not inspect element) on your key pages.
- Confirm the main copy, headings and internal links are all there.
- If they are not, move rendering to the server or add prerendered snapshots.
The View Rendered Source browser extension (ALT + U) shows raw versus rendered HTML side by side, with green highlights for anything JavaScript added.
Also check for content behind logins, paywalls or interaction gates. Crawlers cannot navigate those.
Step 5: Make pages easy to extract and cite
Meta has published no citation ranking signals, so what follows is cross-platform practice. It is cheap, it helps humans, and it is unlikely to hurt.
Lead with the answer
Put a direct answer near the top of each page or section, under a heading that matches a real question. Duda's guide cites Conductor's finding that this answer-first model is more likely to be featured in generative summaries. Hedged copy is harder to quote, so make direct claims where you have the evidence.
Write self-contained passages
Fastlook's guide notes that AI systems struggle with pronoun-heavy writing and paragraphs that need surrounding context. Write passages that make sense on their own, with the entity named, not "it" or "this tool".
Use structure the machine can read
- A clean heading hierarchy and semantic HTML elements
- Accurate image alt text
- JSON-LD structured data that matches the visible content (Google's guidance is explicit on that last point)
- Descriptive title tags, since LLMs read them when crawling and when citing
Back claims with data and attribution
Specific numbers with visible sources are more quotable than generalities. Credible citations, expert quotes and case studies give a model something to lift.
Keep things fresh
Update dates and visible revisions help, and Promptwatch's content half-life data shows that in ChatGPT, news pages lost half their cited pages within 36 days while landing pages and docs stayed cited longest. That is ChatGPT data, not Meta AI, but it is a reasonable hint about which page types hold value.
Step 6: Fix the technical basics
| Area | What to check | Why it matters |
|---|---|---|
| Site architecture | Parent and child pages, topic clusters, internal links | The more internal links a page has, the likelier a crawler finds it |
| URLs | Canonicals, controlled parameters, faceted navigation limits | Prevents crawl waste on near-duplicates |
| Sitemaps | Current XML sitemap with accurate lastmod | Signals freshness and recrawl priority |
| Speed | Fast server response, key content high in the HTML | AI systems often operate with 1 to 5 second timeouts |
| Errors | Redirects for moved pages, few 404s | AI crawlers waste budget on dead URLs |
The timeout figure comes from Simple Machines Marketing as cited in the research; treat it as a rough benchmark rather than a measured Meta behavior.
For audits, Semrush One's Site Audit flags crawlability and rendering issues, and Lumar and Botify cover the enterprise end.
Also consider llms.txt. Duda generates one automatically for its sites, and llmtxt.info has a free generator and validator. I have no evidence Meta reads it, so treat it as low-cost optional, not a Meta lever.
Step 7: Measure whether Meta AI cites you
This is the weak spot. I found no published data on Meta AI citations, and Meta provides no webmaster report for them. What you can do:
- Track crawler hits from Meta user agents over time, as above.
- Ask Meta AI your target questions manually and note whether your domain is cited.
- Use a visibility tracker that covers Meta AI. Promptwatch monitors the real Meta AI interface, alongside ChatGPT, Google AI Overviews, Gemini, Perplexity and others, and ties crawler activity to citations and AI-referred traffic.
- Compare crawl to citation. Being crawled does not mean being cited, so pages that Meta visits but never cites are your content or relevance problem.
For a wider selection of options, the GEO software directory at bestgeosoftware.com lists tools by feature. Others in our catalog that track AI visibility include Profound, Peec AI and Otterly.AI.
A quick checklist
- Decide your policy for Meta: citations yes or no, training yes or no.
- Edit robots.txt to match, then wait 24 hours.
- Check CDN and WAF rules for accidental blocks.
- Search logs for the four Meta user agents.
- Cache and rate limit if load climbs.
- Confirm key content appears in raw HTML.
- Add direct answers, clean headings and JSON-LD to priority pages.
- Fix 404s and redirects, and keep the sitemap current.
- Query Meta AI manually each month and record the results.
What we do not know
I would rather say this plainly than pad the guide. Nobody outside Meta has published how Meta AI chooses sources. We do not know if its own index has launched, whether Meta-WebIndexer fully honors robots.txt, or whether the crawlers execute JavaScript. The crawler surge is well documented. The reason behind it and what it means for citations are not. Get the access and HTML basics right now, watch your logs, and revisit when Meta publishes more.
For broader AI search data across platforms, see the research hub at Promptwatch Data.