Key takeaways
- If your docs already live on Mintlify, Fern, GitBook or ReadMe, the built-in llms.txt generation is the right pick. Don't bolt a second tool on top.
- For live sites with no docs platform, a crawler-based generator gives you a first draft. Plan to trim it hard, because a dump of every URL defeats the point of the file.
- Validators are the underrated part of the stack. Check for broken links, missing descriptions and wrong content types after every docs deploy.
- llms.txt and .md pages are most useful for coding agents fetching developer docs. The evidence that they move consumer AI search citations is thin to nonexistent.
- Measure the result with crawler logs and citation tracking, not with a green checkmark from a generator.
What llms.txt is, in two paragraphs
llms.txt is a markdown file served at /llms.txt. Jeremy Howard proposed it on September 3, 2024 at llmstxt.org. The format is simple: an H1 with the site or project name, a blockquote summary, and H2 sections listing links to the pages a model should read, each with a short description. Companion files exist too. llms-full.txt carries expanded content, and individual pages can offer clean markdown by appending .md to the URL.
It borrows its location convention from robots.txt and sitemap.xml, but it does a different job. robots.txt says what crawlers may fetch. sitemap.xml lists indexable pages. llms.txt is meant to be read at inference time, when an assistant or coding agent is trying to answer a question about your product right now. It is a proposed convention, not a W3C or IETF standard, and the better generators say so plainly.
The short answer: does it help developer docs?
Sort of, and the "sort of" matters. Here is how I would split it.
Developer docs and coding agents: probably yes. Agents like Claude Code and Codex fetch documentation directly, and markdown is cheaper to parse and keeps the context window clean. Promptwatch's Markdown in AI Search report (Feb 2026, based on the prior 7 days of citations) saw the top requested paths on the sampled sites all be /docs/.../*.md, mostly hit by anthropic-claudebot and openai-searchbot. That sample is small and the paths are masked, so treat it as a signal and not a benchmark. It also did not measure llms.txt itself.
Consumer AI search citations: no evidence of a lift. The same report found HTML makes up 99.94% of AI search citations, markdown 0.05%, roughly 2,000 to 1. Its conclusion: markdown does not influence citations in AI search. Put the effort into heading hierarchy, semantic HTML and direct answers, and fix client-side rendering first.
There are also skeptical third-party claims floating around. An Ahrefs study of 137,000+ domains reportedly found about 97% of llms.txt files got zero AI crawler requests, and Google's John Mueller reportedly said the file has no effect for any crawler or LLM. Both reached me as secondhand summaries, so check the originals before quoting them. They do match what the first-party data suggests, though.
My read: publish llms.txt if it costs you nothing (and on a docs platform it does), keep .md endpoints for your docs, and don't expect it to rescue a marketing site's AI visibility.
Generators at a glance
The tools fall into four groups: docs platforms that emit the file as part of the build, static site generator plugins, standalone crawlers, and API-first services.
| Tool | Type | Generates llms-full.txt | Cost | Best for |
|---|---|---|---|---|
| Mintlify | Docs platform | Yes | Free tier | Hosted docs, zero config |
| Fern | Docs platform | Yes | From $400/mo | API-heavy docs, serves markdown to LLM bots |
| GitBook | Docs platform | Yes | Free tier | Teams already on GitBook |
| ReadMe | Docs platform | No | Free tier | Toggle-on simplicity |
| Context.dev | API | Yes | Free tier, usage-based | Programmatic generation at scale |
| llmstxtgenerate.com | Web form + validator | Yes ($1 one-time) | Free tier | Non-technical users who also want validation |
| llmstxtgen.com | Web form | Yes | Free, 20-page cap | Small sites |
| Apify actor | Crawler | llms.txt only | Pay-per-use | Existing Apify users |
| Firecrawl | Crawler (deprecated) | Yes | n/a | Legacy use only |
Pricing and limits come from the vendors' comparison pages and change often. Confirm on each site before committing.
Docs platforms that generate it for you
If the source of truth is already a docs platform, this is the easy route.
Mintlify
Mintlify auto-generates /llms.txt, /llms-full.txt and .md versions of every page for all hosted docs, with no configuration. It rolled this out in November 2024, which handed the file to thousands of docs sites overnight, including Anthropic, Cursor, Pinecone and Windsurf. The limitation is scope: it only works for content hosted there, and you get little control over what is included or analytics on whether anyone reads it.

To be fair, Mintlify wrote that comparison page and naturally ranks itself first. The tables are still useful as a map of the options.
Fern, GitBook, ReadMe and Redocly
Fern auto-enables llms.txt and llms-full.txt and serves markdown to LLM bots, starting at $400/mo. GitBook generates both files with no setup. ReadMe generates llms.txt only, with a toggle and .md via URL suffix. Redocly has a toggle for llms.txt and no llms-full.txt. Pick based on the platform you already run, not on this feature alone.
Static site generator plugins
If your docs are built from a repo, a build-time plugin keeps the file in sync with the source, which beats a crawler that guesses.
| Framework | Plugin | llms.txt | llms-full.txt | .md versions |
|---|---|---|---|---|
| Docusaurus | docusaurus-plugin-llms | Yes | Yes | Yes |
| Astro Starlight | starlight-llms-txt | Yes | Yes | Yes (plus llms-small.txt) |
| VitePress | vitepress-plugin-llms | Yes | Yes | Yes |
| Sphinx / Read the Docs | sphinx-llms-txt | Yes | Yes | No |
| MkDocs | mkdocs-llmstxt | Yes | n/a | n/a |
| Hugo | Custom output format | Yes | Manual | No |
I did not test these plugins myself, and the table comes from Mintlify's roundup, so check each plugin's repo for maintenance status before adopting it. The MkDocs plugin generates llms.txt from source files during the build, which is the right model: the file changes when your docs change.
Standalone and API-first generators
Context.dev
Context.dev is an API, not a web form. You crawl a site, get clean markdown back, and assemble llms.txt and llms-full.txt in your own pipeline. That makes sense if you generate files for many domains on a schedule. It makes no sense if you need one file once. It has no built-in spec validator or readiness score, so pair it with a validator.

Context.dev publishes that comparison and puts itself on top, so read it with that in mind.
llmstxtgenerate.com and llmstxtgen.com
llmstxtgenerate.com accepts a sitemap, a URL it crawls, or a pasted list of URLs. The free tier gives titles, links and descriptions. A $1 one-time payment unlocks llms-full.txt, with either summaries or full content, for 25 to 250 pages. Its validator is the reason to use it: it flags broken links, missing descriptions and structural errors, and returns an AI Readiness Score out of 100. Uploaded files are read locally in the browser. llmstxtgen.com is free, needs no signup and caps at 20 pages, so it suits a small site that needs a draft in a minute.
Firecrawl, Apify and the rest
Firecrawl's hosted llms.txt tool was deprecated in June 2025. The repo points to a create-llmstxt-py example instead. It used GPT-4o-mini to write page titles and descriptions, which worked well, but it is unmaintained now. The Apify llmstxt-generator actor deep-crawls with Crawlee, with a 4 GB memory cap that can slow big crawls. Other free options named in roundups include LLMTEXT from Parallel.ai (sitemap-based, with a validator and MCP converter), SiteSpeakAI and llms-txt.io. On the WordPress side, All in One SEO emits the file.
For a quick browser-based option, the catalog also lists llmtxt.info, a free generator and validator.

Validators: what to check
Generators get the attention, but validation catches the failures that actually hurt. A file can be perfectly formatted and still be useless.
- The file returns 200 at
/llms.txtwith a text/plain or text/markdown content type. Common Crawl's July 2026 analysis dropped 14,191 empty responses out of 598,298 text/plain or text/markdown hits, so empty files are a real thing. - Every link resolves. Broken links are the most common failure after a docs reorganization.
- Every link has a short description a model can use to decide whether to fetch it.
- The file changed after your last docs deploy. A stale hash means your build step is not regenerating it.
- The file is curated. LLM Pulse warns that sitemap-dump generators produce 5,000-line files with every legal page, and recommends trimming a first draft by 30 to 70 percent. I agree with that completely.
llmstxtgenerate.com and LLMTEXT both include validators. For a repeatable check, add a CI step that fetches the file, parses the links and fails the build on any non-200.
# minimal link check for a generated llms.txt
curl -s https://example.com/llms.txt \
| grep -oE 'https?://[^) ]+' \
| sort -u \
| while read url; do
code=$(curl -s -o /dev/null -w '%{http_code}' "$url")
[ "$code" != "200" ] && echo "$code $url"
done
How to pick
- On a docs platform already? Use its built-in generation. Done.
- Docs built from a repo? Use the build plugin for your framework so the file regenerates with every build.
- Live site, no docs platform? Run a crawler-based generator, then hand-edit the output down to the pages that matter.
- Many sites, scheduled runs? Use an API such as Context.dev and add your own validation step.
- One small static site? Write it by hand. One honest file beats a generated one you never look at.
Check whether anything is actually reading it
The part most guides skip is measurement. Publishing the file proves nothing. You want to know whether AI crawlers fetch it, which docs pages they read, and whether any of that leads to citations or visits.
IP-verified crawler logs are the cleanest way to see this. Tools like Promptwatch show which AI bots hit which pages, what errors they hit, and whether those pages later get cited. Its AI crawler logs can tell you if /llms.txt or your .md pages are being requested at all, so you can separate an access problem from a content problem.

Promptwatch's AI Crawler Traffic report (verified requests, June 8 to September 6, 2026) is a useful baseline here. OpenAI made up 79.8% of verified AI crawler requests in the week of August 31 to September 6, down from 94.8% in early June. Its advice applies directly: check robots.txt, CDN and WAF rules for each provider, and compare your logs with that mix. A provider that is large in the report but near zero in your logs likely cannot reach you.
DarkVisitors is another option if you only want to see agents and bots visiting.

For a broader look at the category, the GEO software directory at bestgeosoftware.com lists visibility and tracking platforms.
What to do on developer docs, concretely
- Keep
.mdendpoints for every docs page. Coding agents fetch them directly. - Publish llms.txt with sections that mirror your docs navigation: quickstart, API reference, guides, changelog.
- Add an
llms-full.txtonly if your docs are small enough for it to be useful. A multi-megabyte file helps nobody. - Make sure your CDN does not block ClaudeBot, OAI-SearchBot or other AI user agents on
/docs. - For marketing and blog pages, skip the
.mdmirrors. Spend the time on HTML structure and answer-first writing instead, which is what the data says drives citations. - Re-check crawler logs a month after launch. If nothing fetches the file, you have learned something useful for free.
If you also want to see whether Lighthouse or other tooling audits for the file, note that the llmstxt.org page now says Chrome Lighthouse audits for it, according to a US Tech Automations summary of the spec. I could not verify that independently.
Bottom line
Pick the generator that matches where your content lives, validate on every deploy, and be realistic about the payoff. For developer docs, llms.txt and clean markdown are cheap insurance that agents can read you. For consumer AI search, the citation data points at HTML quality instead. Generate the file, then watch your crawler logs to see if it earns its keep.