llms.txt: what it does, what it does not do, and how to publish it
A factual implementation guide to the voluntary llms.txt proposal, including scope, format, maintenance, and common mistakes.
Direct answer
llms.txt is a voluntary proposal for publishing a concise, Markdown-formatted guide to important site content at /llms.txt. It can help an LLM-oriented client discover authoritative pages, but it is not an access-control file, does not replace robots.txt or a sitemap, and does not guarantee indexing or citation.
Understand the status of llms.txt
The llms.txt project describes itself as a proposal. Site owners can publish the file today, but support varies by product and should not be assumed. Its useful role is documentation: explain the site and point to a curated set of stable, authoritative resources.
Crawler permissions still belong in robots.txt. Canonical discovery still depends on ordinary links, sitemaps, HTTP responses, and page metadata. Calling llms.txt a permission grammar or a universal ranking signal overstates what the file currently does.
Publish a small, curated map
Place a UTF-8 Markdown document at the root path /llms.txt. Start with one H1, add a short site summary, and group the most useful links under descriptive H2 headings. Each link should include enough context for a client to decide whether the page is relevant.
Prefer canonical absolute URLs so the file is portable outside the site. Include product documentation, key explanatory guides, pricing, and trust pages when they are important to understanding the business. Exclude login screens, duplicate URLs, temporary campaigns, and thin pages.
- Use the public brand or product name as the H1.
- Describe what the organization provides without marketing superlatives.
- Link only to successful, indexable, canonical URLs.
- Keep descriptions factual and synchronized with the linked page.
- Offer a larger llms-full.txt only when the additional text is maintained.
Treat maintenance as the real implementation
A file that lists retired products, old prices, or redirected URLs can create conflicting machine-readable facts. Generate it from the same content catalogue or route data used by the site whenever possible, then test every referenced URL during deployment.
Monitor access logs if your infrastructure exposes them, but do not infer broad model adoption from a small number of requests. The more reliable success criteria are correctness, low maintenance cost, and zero contradiction with the human-readable site.
Verify the endpoint
Request the exact root URL and confirm a 200 response, a text content type, readable Markdown, and no authentication challenge. Then follow each link and compare names, descriptions, prices, and update dates with the corresponding page.
Implementation checklist
- ✓/llms.txt returns 200 without authentication.
- ✓The file explicitly says it follows a voluntary proposal.
- ✓All linked URLs are absolute, canonical, public, and successful.
- ✓Product descriptions and prices match the visible site.
- ✓robots.txt and sitemap.xml remain the sources for access and URL discovery.
- ✓A deployment check detects dead or redirected links in the file.
Frequently asked questions
Does llms.txt replace robots.txt?
No. robots.txt is the standardized crawler-control mechanism. llms.txt is a proposed content guide and should not be used to express access permissions.
Does Google or OpenAI guarantee support for llms.txt?
No universal support guarantee exists. Publish it as a low-cost, machine-readable aid, not as a substitute for indexable pages and established discovery mechanisms.
Should llms.txt list every page?
Usually not. Its proposed format is best suited to a curated set of high-value resources with short descriptions. A sitemap is the appropriate format for broad URL enumeration.
Primary sources
These references support the standards and product behavior described above. They do not imply endorsement of Index Instrument.