Best llms.txt Generators and Validators in 2026 (and Whether They Help Developer Docs)

A practical comparison of llms.txt generators, docs-platform auto-generation, SSG plugins and validators in 2026, plus an honest look at whether the file helps developer docs or consumer AI search.

Key takeaways

  • If your docs already live on Mintlify, Fern, GitBook or ReadMe, the built-in llms.txt generation is the right pick. Don't bolt a second tool on top.
  • For live sites with no docs platform, a crawler-based generator gives you a first draft. Plan to trim it hard, because a dump of every URL defeats the point of the file.
  • Validators are the underrated part of the stack. Check for broken links, missing descriptions and wrong content types after every docs deploy.
  • llms.txt and .md pages are most useful for coding agents fetching developer docs. The evidence that they move consumer AI search citations is thin to nonexistent.
  • Measure the result with crawler logs and citation tracking, not with a green checkmark from a generator.

What llms.txt is, in two paragraphs

llms.txt is a markdown file served at /llms.txt. Jeremy Howard proposed it on September 3, 2024 at llmstxt.org. The format is simple: an H1 with the site or project name, a blockquote summary, and H2 sections listing links to the pages a model should read, each with a short description. Companion files exist too. llms-full.txt carries expanded content, and individual pages can offer clean markdown by appending .md to the URL.

It borrows its location convention from robots.txt and sitemap.xml, but it does a different job. robots.txt says what crawlers may fetch. sitemap.xml lists indexable pages. llms.txt is meant to be read at inference time, when an assistant or coding agent is trying to answer a question about your product right now. It is a proposed convention, not a W3C or IETF standard, and the better generators say so plainly.

The short answer: does it help developer docs?

Sort of, and the "sort of" matters. Here is how I would split it.

Developer docs and coding agents: probably yes. Agents like Claude Code and Codex fetch documentation directly, and markdown is cheaper to parse and keeps the context window clean. Promptwatch's Markdown in AI Search report (Feb 2026, based on the prior 7 days of citations) saw the top requested paths on the sampled sites all be /docs/.../*.md, mostly hit by anthropic-claudebot and openai-searchbot. That sample is small and the paths are masked, so treat it as a signal and not a benchmark. It also did not measure llms.txt itself.

Consumer AI search citations: no evidence of a lift. The same report found HTML makes up 99.94% of AI search citations, markdown 0.05%, roughly 2,000 to 1. Its conclusion: markdown does not influence citations in AI search. Put the effort into heading hierarchy, semantic HTML and direct answers, and fix client-side rendering first.

There are also skeptical third-party claims floating around. An Ahrefs study of 137,000+ domains reportedly found about 97% of llms.txt files got zero AI crawler requests, and Google's John Mueller reportedly said the file has no effect for any crawler or LLM. Both reached me as secondhand summaries, so check the originals before quoting them. They do match what the first-party data suggests, though.

My read: publish llms.txt if it costs you nothing (and on a docs platform it does), keep .md endpoints for your docs, and don't expect it to rescue a marketing site's AI visibility.

Generators at a glance

The tools fall into four groups: docs platforms that emit the file as part of the build, static site generator plugins, standalone crawlers, and API-first services.

ToolTypeGenerates llms-full.txtCostBest for
MintlifyDocs platformYesFree tierHosted docs, zero config
FernDocs platformYesFrom $400/moAPI-heavy docs, serves markdown to LLM bots
GitBookDocs platformYesFree tierTeams already on GitBook
ReadMeDocs platformNoFree tierToggle-on simplicity
Context.devAPIYesFree tier, usage-basedProgrammatic generation at scale
llmstxtgenerate.comWeb form + validatorYes ($1 one-time)Free tierNon-technical users who also want validation
llmstxtgen.comWeb formYesFree, 20-page capSmall sites
Apify actorCrawlerllms.txt onlyPay-per-useExisting Apify users
FirecrawlCrawler (deprecated)Yesn/aLegacy use only

Pricing and limits come from the vendors' comparison pages and change often. Confirm on each site before committing.

Docs platforms that generate it for you

If the source of truth is already a docs platform, this is the easy route.

Mintlify

Mintlify auto-generates /llms.txt, /llms-full.txt and .md versions of every page for all hosted docs, with no configuration. It rolled this out in November 2024, which handed the file to thousands of docs sites overnight, including Anthropic, Cursor, Pinecone and Windsurf. The limitation is scope: it only works for content hosted there, and you get little control over what is included or analytics on whether anyone reads it.

Mintlify's guide comparing llms.txt implementation platforms, plugins and standalone generators

To be fair, Mintlify wrote that comparison page and naturally ranks itself first. The tables are still useful as a map of the options.

Fern, GitBook, ReadMe and Redocly

Fern auto-enables llms.txt and llms-full.txt and serves markdown to LLM bots, starting at $400/mo. GitBook generates both files with no setup. ReadMe generates llms.txt only, with a toggle and .md via URL suffix. Redocly has a toggle for llms.txt and no llms-full.txt. Pick based on the platform you already run, not on this feature alone.

Static site generator plugins

If your docs are built from a repo, a build-time plugin keeps the file in sync with the source, which beats a crawler that guesses.

FrameworkPluginllms.txtllms-full.txt.md versions
Docusaurusdocusaurus-plugin-llmsYesYesYes
Astro Starlightstarlight-llms-txtYesYesYes (plus llms-small.txt)
VitePressvitepress-plugin-llmsYesYesYes
Sphinx / Read the Docssphinx-llms-txtYesYesNo
MkDocsmkdocs-llmstxtYesn/an/a
HugoCustom output formatYesManualNo

I did not test these plugins myself, and the table comes from Mintlify's roundup, so check each plugin's repo for maintenance status before adopting it. The MkDocs plugin generates llms.txt from source files during the build, which is the right model: the file changes when your docs change.

Standalone and API-first generators

Context.dev

Context.dev is an API, not a web form. You crawl a site, get clean markdown back, and assemble llms.txt and llms-full.txt in your own pipeline. That makes sense if you generate files for many domains on a schedule. It makes no sense if you need one file once. It has no built-in spec validator or readiness score, so pair it with a validator.

Context.dev's comparison of llms.txt generator tools in 2026

Context.dev publishes that comparison and puts itself on top, so read it with that in mind.

llmstxtgenerate.com and llmstxtgen.com

llmstxtgenerate.com accepts a sitemap, a URL it crawls, or a pasted list of URLs. The free tier gives titles, links and descriptions. A $1 one-time payment unlocks llms-full.txt, with either summaries or full content, for 25 to 250 pages. Its validator is the reason to use it: it flags broken links, missing descriptions and structural errors, and returns an AI Readiness Score out of 100. Uploaded files are read locally in the browser. llmstxtgen.com is free, needs no signup and caps at 20 pages, so it suits a small site that needs a draft in a minute.

Firecrawl, Apify and the rest

Firecrawl's hosted llms.txt tool was deprecated in June 2025. The repo points to a create-llmstxt-py example instead. It used GPT-4o-mini to write page titles and descriptions, which worked well, but it is unmaintained now. The Apify llmstxt-generator actor deep-crawls with Crawlee, with a 4 GB memory cap that can slow big crawls. Other free options named in roundups include LLMTEXT from Parallel.ai (sitemap-based, with a validator and MCP converter), SiteSpeakAI and llms-txt.io. On the WordPress side, All in One SEO emits the file.

For a quick browser-based option, the catalog also lists llmtxt.info, a free generator and validator.

Favicon of llmtxt.info

llmtxt.info

Free llms.txt generator and validator
View more
Screenshot of llmtxt.info website

Validators: what to check

Generators get the attention, but validation catches the failures that actually hurt. A file can be perfectly formatted and still be useless.

  • The file returns 200 at /llms.txt with a text/plain or text/markdown content type. Common Crawl's July 2026 analysis dropped 14,191 empty responses out of 598,298 text/plain or text/markdown hits, so empty files are a real thing.
  • Every link resolves. Broken links are the most common failure after a docs reorganization.
  • Every link has a short description a model can use to decide whether to fetch it.
  • The file changed after your last docs deploy. A stale hash means your build step is not regenerating it.
  • The file is curated. LLM Pulse warns that sitemap-dump generators produce 5,000-line files with every legal page, and recommends trimming a first draft by 30 to 70 percent. I agree with that completely.

llmstxtgenerate.com and LLMTEXT both include validators. For a repeatable check, add a CI step that fetches the file, parses the links and fails the build on any non-200.

# minimal link check for a generated llms.txt
curl -s https://example.com/llms.txt \
  | grep -oE 'https?://[^) ]+' \
  | sort -u \
  | while read url; do
      code=$(curl -s -o /dev/null -w '%{http_code}' "$url")
      [ "$code" != "200" ] && echo "$code $url"
    done

How to pick

  1. On a docs platform already? Use its built-in generation. Done.
  2. Docs built from a repo? Use the build plugin for your framework so the file regenerates with every build.
  3. Live site, no docs platform? Run a crawler-based generator, then hand-edit the output down to the pages that matter.
  4. Many sites, scheduled runs? Use an API such as Context.dev and add your own validation step.
  5. One small static site? Write it by hand. One honest file beats a generated one you never look at.

Check whether anything is actually reading it

The part most guides skip is measurement. Publishing the file proves nothing. You want to know whether AI crawlers fetch it, which docs pages they read, and whether any of that leads to citations or visits.

IP-verified crawler logs are the cleanest way to see this. Tools like Promptwatch show which AI bots hit which pages, what errors they hit, and whether those pages later get cited. Its AI crawler logs can tell you if /llms.txt or your .md pages are being requested at all, so you can separate an access problem from a content problem.

Favicon of Promptwatch

Promptwatch

Track and optimize your brand's visibility in AI search engines
View more
Screenshot of Promptwatch website

Promptwatch's AI Crawler Traffic report (verified requests, June 8 to September 6, 2026) is a useful baseline here. OpenAI made up 79.8% of verified AI crawler requests in the week of August 31 to September 6, down from 94.8% in early June. Its advice applies directly: check robots.txt, CDN and WAF rules for each provider, and compare your logs with that mix. A provider that is large in the report but near zero in your logs likely cannot reach you.

DarkVisitors is another option if you only want to see agents and bots visiting.

Favicon of DarkVisitors

DarkVisitors

Track AI agents, bots, and LLM referrals visiting your websi
View more
Screenshot of DarkVisitors website

For a broader look at the category, the GEO software directory at bestgeosoftware.com lists visibility and tracking platforms.

What to do on developer docs, concretely

  • Keep .md endpoints for every docs page. Coding agents fetch them directly.
  • Publish llms.txt with sections that mirror your docs navigation: quickstart, API reference, guides, changelog.
  • Add an llms-full.txt only if your docs are small enough for it to be useful. A multi-megabyte file helps nobody.
  • Make sure your CDN does not block ClaudeBot, OAI-SearchBot or other AI user agents on /docs.
  • For marketing and blog pages, skip the .md mirrors. Spend the time on HTML structure and answer-first writing instead, which is what the data says drives citations.
  • Re-check crawler logs a month after launch. If nothing fetches the file, you have learned something useful for free.

If you also want to see whether Lighthouse or other tooling audits for the file, note that the llmstxt.org page now says Chrome Lighthouse audits for it, according to a US Tech Automations summary of the spec. I could not verify that independently.

Bottom line

Pick the generator that matches where your content lives, validate on every deploy, and be realistic about the payoff. For developer docs, llms.txt and clean markdown are cheap insurance that agents can read you. For consumer AI search, the citation data points at HTML quality instead. Generate the file, then watch your crawler logs to see if it earns its keep.

Share:

© 2026 AI Search Tools · Best AI search tools and platforms · RSS

AI Search Tools is an affiliate review site. When you click links to vendors or buy through links on our site, we may earn an affiliate commission at no extra cost to you.

AI Search Tools is a review website based on user reviews on Reddit and G2, and on publicly available information. We keep everything as up to date as possible, but pricing and features can change. Always confirm the details with the vendor before purchasing.