How to Get Indexed by AI Through llms.txt: The Practical Workflow

How to get indexed by LLM through llms.txt starts with a plain-text file at your root domain that lists your most important pages for machines. It is not a magic switch. It is a hint file, similar in spirit to robots.txt or sitemap.xml, that tells language models which URLs matter.

The spec is young. No major AI vendor has publicly promised to read it. But early adopters report it costs almost nothing to ship, and it forces you to organize your site the way an LLM wants to see it.

This guide walks through the file format, the hosting rules, the verification checks that prove crawlers actually fetched it, and the fallback tactics that keep working when a model ignores your file.

What llms.txt Actually Is

llms.txt is a plain-text Markdown file you place at the root of your domain to tell language models which pages best represent your site. The format was proposed in 2024 by Jeremy Howard and is documented at llmstxt.org. It is a machine-readable summary, not a permission file.

The file uses Markdown headings and bullet lists. Each bullet is a link with a short description. Models that support the spec can fetch it once and use it as a map of your site.

• Location: https://yourdomain.com/llms.txt — root of the primary domain
• Format: Markdown, UTF-8, no login wall, served as text/plain or text/markdown
• Purpose: point LLMs at your best pages with short human-readable descriptions
• Optional companion: llms-full.txt bundles the full text of key pages into one file
• Not a directive: crawlers can ignore it, unlike robots.txt which most respect

The Exact File Structure to Publish

A working llms.txt file starts with an H1 for your site name, a short blockquote summary, and then H2 sections that group your best URLs as bulleted links. Keep it under a few hundred lines so a model can read the whole file in one context window.

The structure below matches the reference examples on llmstxt.org and the version Anthropic publishes for its docs. Copy the shape exactly.

• H1 line: # Your Company Name
• Blockquote: > One-sentence description of what your site covers
• Optional intro paragraph (2-3 sentences of context)
• H2 sections like ## Docs, ## Guides, ## Products, ## About
• Under each H2, a bullet list of links: - [Page title](https://url): short description
• Optional ## Optional section at the end for secondary pages a model can skip

Hosting Requirements for llms.txt Indexing

For llms.txt indexing to work, the file must live at the root of your primary domain and load over HTTPS with a 200 response and no authentication. If the file 404s, redirects, or hides behind Cloudflare bot protection, most crawlers will skip it.

The hosting rules are simple but easy to get wrong on modern static hosts and CDNs. Check each item below before you consider the file live.

• URL: https://yourdomain.com/llms.txt — not on a subdomain, not under /docs/
• Status: HTTP 200, no redirect chain
• Content-Type: text/plain; charset=utf-8 or text/markdown
• Access: no login, no paywall, no aggressive bot challenge
• Size: under 100 KB is safe; a full llms-full.txt can go larger
• CORS: not required for crawlers, but harmless to allow

Verification Checks: Did the File Actually Get Pulled

The only reliable way to know llms.txt indexing worked is to check your server logs for known AI crawler user agents fetching the file. A 200 in a browser proves nothing about whether a language model read it.

Filter your access logs for the user agents below over the last 7 to 30 days. If none of them requested /llms.txt, the file is live but unread, and you should focus on the fallback tactics later in this guide.

• GPTBot — OpenAI's documented crawler
• OAI-SearchBot — OpenAI's search-specific bot for ChatGPT search results
• ChatGPT-User — fired when a ChatGPT user asks the model to fetch a URL live
• Google-Extended — Google's opt-in signal for Gemini training data
• PerplexityBot and Perplexity-User — Perplexity's crawl and on-demand fetch
• ClaudeBot, Claude-Web, anthropic-ai — Anthropic's crawlers
• CCBot — Common Crawl, which feeds many open models
• Bytespider — ByteDance/Doubao

Which Crawlers Actually Respect llms.txt

As of 2026, no major AI vendor has publicly committed to reading llms.txt as a ranking signal, so llms.txt indexing remains an unofficial channel. The most-cited real adopters are documentation tools and framework maintainers, not the flagship models.

That said, the file is trivially discoverable at a well-known path. Some crawlers may fetch it opportunistically even without a public policy. Ship it, verify with logs, but do not treat it as your primary AI-visibility play.

• Confirmed support: Cursor, Windsurf, Zed and other AI IDEs use llms.txt to load docs into context
• Publishing but not confirmed reading: Anthropic, Cloudflare, Hugging Face, and many docs sites publish llms.txt
• No public commitment: OpenAI, Google Gemini, Perplexity, and Anthropic have not confirmed the file is used for indexing or ranking
• Practical read: treat llms.txt as helpful hygiene, not a substitute for real citations and structured content

Step-by-Step llms.txt Setup

The full llms.txt setup takes about an hour for a small site and maps closely to how you would build a curated sitemap for humans. Work through the steps below in order so you catch hosting mistakes before you announce the file.

The goal is a file that a model can skim in one pass and come away with an accurate picture of what your site is best at.

• 1. List your 20 to 50 highest-value URLs — pillar guides, product pages, pricing, docs
• 2. Write a 1-sentence description for each link, plain-English, no marketing fluff
• 3. Group the links under H2 section headings by topic or content type
• 4. Add the H1 site name and blockquote summary at the top
• 5. Save as llms.txt, UTF-8, and deploy to the site root
• 6. Curl the URL from a clean environment: curl -I https://yourdomain.com/llms.txt
• 7. Confirm 200, correct Content-Type, and no redirect
• 8. Submit the URL to Google Search Console and Bing Webmaster Tools as a normal page

Fallback Tactics if the Engine Ignores llms.txt

If your logs show no AI crawler ever fetched llms.txt, the file is not your problem — the model simply is not using it. Fall back to the tactics that measurably move AI visibility today: clean structured content, third-party citations, and being present where models actually look.

These tactics work whether llms.txt is respected or not, so many teams treat llms.txt as insurance and put their real effort here.

• Publish a clean HTML sitemap.xml and keep robots.txt permissive to AI bots you want
• Add Schema.org Article, FAQPage, and Organization structured data
• Earn citations on sites LLMs already trust: Wikipedia, high-authority publications, Reddit threads relevant to your topic
• Answer the exact questions users ask, in the first sentence of each section (AEO extractability)
• Monitor mentions in Perplexity and ChatGPT with tools like Profound, Peec.ai, or Ahrefs Brand Radar
• Publish a companion llms-full.txt with the full text of your best pages for models that support it

Verdict: Ship It, But Do Not Bet On It

How to get indexed by LLM through llms.txt is worth an hour of work and no more, because the spec is young and unenforced. Ship the file, verify with server logs, and move on to the tactics that AI models already reward.

Treat llms.txt as a low-cost bet that pays off if the standard catches on. The real AI-visibility wins in 2026 still come from structured content, third-party citations, and answering the exact questions your buyers ask.

• Cost to ship: about 1 hour for a small site
• Downside risk: near zero
• Upside: model-dependent, currently unproven for the big four LLMs
• Do not skip: structured data, real citations, and AEO-friendly writing