llms.txt Explained: What It Is, Whether It Works, and When to Ship One

llms.txt is a proposed plain-text file placed at the root of a website that gives large language models a curated map of your most important content. It borrows the spirit of robots.txt and sitemap.xml, but its audience is AI reasoning systems, not crawlers.

The proposal has spread fast across the answer engine optimization community, but the actual adoption picture is messier than the hype suggests. Some tools consume it, most major AI search engines do not, and the file is easy to misuse.

This guide walks through what llms.txt is, what the spec actually requires, which engines respect it as of 2026, when shipping one helps, when it is overkill, and the common mistakes we see teams make.

What Is llms.txt?

llms.txt is a Markdown file served at the root of a domain that lists the pages and resources you want large language models to prioritize when they answer questions about your site. The proposal was published by Jeremy Howard of Answer.AI in September 2024 and has since been adopted by a growing set of documentation sites and SaaS vendors.

The core idea is simple. LLM context windows are limited, HTML pages are noisy, and marketing wrappers waste tokens. A curated Markdown index lets a model reach the important content without crawling your whole site or parsing navigation.

It is a proposal, not a standard. There is no W3C or IETF working group behind it, and no major search engine has committed to consuming it in production ranking or answer generation.

• Lives at yourdomain.com/llms.txt as plain Markdown
• Human-readable and machine-parseable in the same file
• Optional companion file llms-full.txt bundles the actual content
• Modeled after the readability goals of robots.txt and sitemap.xml
• Authored by Jeremy Howard, published September 2024

The llms.txt Spec: What It Actually Requires

The llms.txt spec defines a strict Markdown structure with an H1 site name, a blockquote summary, optional context paragraphs, and H2 section headings that group links. Each link is a bullet with an optional description after a colon.

The file must be valid Markdown so both humans and models can parse it without a custom reader. The spec also defines optional companion files, most notably llms-full.txt, which inlines the full text of the linked documents.

Sections named Optional at the H2 level signal lower priority. A model with a tight context window can safely skip that section and still answer well.

• H1: the name of the site or project
• Blockquote: one-line description of the project
• Zero or more paragraphs: additional context
• H2 sections: grouped bullet lists of links
• Each bullet: [Title](url): optional description
• Optional H2 Optional: content that can be skipped
• Companion file llms-full.txt inlines full page text

Does llms.txt Actually Work? The Evidence in 2026

As of 2026, no major consumer AI search engine has publicly confirmed that it consumes llms.txt to answer user questions. That includes ChatGPT search, Perplexity, Google AI Overviews, Gemini, Copilot, and Claude.ai search.

In September 2025, John Mueller of Google confirmed on LinkedIn that Google does not use llms.txt for search or AI Overviews. Gary Illyes reiterated the same position in a public appearance later that year.

The tools that do consume llms.txt today are mostly developer-facing: Cursor for documentation lookup, Windsurf, some Anthropic examples, and third-party crawlers used by AEO tracking vendors like Profound and Otterly.ai. Ranking impact for public AI search remains unproven.

• Google: does not use llms.txt (publicly confirmed 2025)
• OpenAI / ChatGPT: no public commitment to consume it
• Perplexity: no public commitment; crawler follows normal HTML
• Anthropic Claude: no ranking use; docs referenced in some examples
• Cursor + Windsurf: consume llms.txt for docs autocomplete
• AEO trackers may parse it as a discovery signal
• No published study proves lift in AI answer citations

When to Ship an llms.txt File

You should ship an llms.txt file when your site is documentation-heavy, when developers integrate with your product through IDEs, or when the marginal cost of publishing one is close to zero. It is a small, cheap signal to send.

The strongest case is technical documentation. Cursor, Windsurf, and Claude Code users regularly point their IDEs at an llms.txt URL to load API reference and how-to content into context. Vendors with clean docs benefit from being easier to reference.

Enterprise and B2B SaaS sites with a clear product story also benefit. A well-structured llms.txt makes it easier for a model to summarize what you do, who you serve, and where to find pricing or contact pages.

• Ship it if you have API or product documentation
• Ship it if developers use your product through AI IDEs
• Ship it if your homepage story is buried under marketing chrome
• Ship it if your competitors have published one and you have not
• Skip it if your site is under 20 pages of shallow content
• Skip it if your team cannot maintain it alongside content updates

When llms.txt Is Overkill

llms.txt is overkill for small marketing sites where your XML sitemap and clean HTML already tell the whole story. If a model can answer questions about your business by reading your homepage, a duplicate Markdown index adds no signal.

It is also overkill for e-commerce catalogs. A 50,000-product store cannot meaningfully list its inventory in one Markdown file, and the spec was never designed for that use case. Structured data and clean product schema do more.

Finally, it is a distraction if your fundamentals are broken. A site that is not indexed, has no citations, and no schema markup will not become visible in AI search because it published one extra file.

• Skip for small marketing sites where the homepage already summarizes you
• Skip for large e-commerce inventories that will not fit
• Skip if you have no clean canonical URLs to link to
• Skip if the file will go stale and never be updated
• Fix fundamentals first: indexing, schema, citations, clean HTML

Common llms.txt Mistakes to Avoid

The most common llms.txt mistake is treating it like a sitemap and dumping every URL on the site into a single flat bullet list. The spec assumes curation. If everything is a priority, nothing is.

The second most common mistake is publishing llms.txt and then never updating it. A stale file that points to old pricing pages, deprecated docs, or dead URLs is worse than no file at all.

Other frequent errors: using the wrong format (HTML instead of Markdown), gating the file behind auth, blocking it in robots.txt, and inlining thousands of words instead of using llms-full.txt for the heavy content.

• Do not dump every URL on the site
• Do not publish once and forget: keep it fresh
• Do not use HTML; the spec requires Markdown
• Do not block /llms.txt in robots.txt
• Do not gate it behind login
• Do not confuse llms.txt with llms-full.txt
• Do not use it as a substitute for a real XML sitemap
• Do not expect ranking lift in Google or Bing from it

llms.txt for SEO: What It Does and Does Not Do

llms.txt for SEO has no confirmed effect on traditional search rankings in Google or Bing. Both search engines rank pages on the open web using their own crawlers and signals, and neither treats the file as a ranking input.

The signal it may send is discovery and disambiguation. AEO tracking tools like Peec.ai and AthenaHQ parse llms.txt when they profile your brand for AI visibility monitoring.

Treat llms.txt as one small piece of an answer engine optimization program, not the whole program. Citations, schema markup, and genuine third-party mentions still do most of the work in AI search visibility.

• No confirmed ranking effect in Google or Bing
• No confirmed citation lift in ChatGPT search or Perplexity
• May help discovery by AEO tracking crawlers
• May help IDE-based developer surfaces (Cursor, Windsurf)
• Not a substitute for citations, schema, or real content

How to Build an llms.txt File the Right Way

The right way to build an llms.txt file is to pick 20 to 50 URLs that best represent what your site is, group them by intent, and describe each in a short sentence. Under 500 lines is a good ceiling for most sites.

Start with your highest-authority explainer pages, your pricing or product page, your contact page, and your best documentation. Group under H2 headings that reflect the actual jobs a reader has: Getting Started, Pricing, Docs, Optional.

Publish at /llms.txt, make sure it returns a 200 response, and register the file in your deploy pipeline so it is updated whenever the underlying pages change. If you have technical docs, add an llms-full.txt companion with the raw Markdown source.

• Pick 20 to 50 curated URLs, not everything
• Group by reader intent under H2 headings
• Write a one-line description per link
• Serve at /llms.txt as plain Markdown, 200 response
• Add llms-full.txt for docs-heavy sites
• Rebuild it whenever you rewrite a linked page
• Test it in Cursor or Claude Code to verify it parses