llms.txt Explained: How to Write, Validate, and Keep One Accurate

Most sites that ship an llms.txt treat it like a sitemap: generate it once, drop it at the root, forget it exists. That instinct is half right. The file is small and mostly static. But unlike a sitemap, nobody's crawler is going to tell you when it's wrong, and the reader it was written for, an AI agent answering a question about your product, will follow a dead or outdated link without complaint.

The format was proposed by Jeremy Howard in September 2024, and the llms.txt specification reached version 2 in August 2026, after two years of real adoption. The v2 changes are telling: the spec dropped the tooling that assumed agents would mechanically expand the file into one big context blob, and rewrote its background to describe "how agents actually use websites" — they read the file, then follow links.

This guide covers what an llms.txt file is, what v2 actually requires, a worked example you can copy, the mistakes a validator catches, how to serve the file so agents find it, and the problem no validator can catch for you: the file still parses, but it no longer tells the truth.

What llms.txt is (and what it is not)#

An llms.txt file is a Markdown document served at /llms.txt that gives a language model a curated map of a site: a one-line summary of what the site is, a few notes an agent needs to avoid mistakes, and lists of links to the pages that matter, each with a short note.

It is easiest to understand next to the files it gets confused with:

The key word in that table is curated. A sitemap lists everything. An llms.txt lists what an agent should read first, in the order a helpful colleague would recommend it. If you want the long version of how these files relate, our AGENTS.md vs CLAUDE.md vs llms.txt guide walks through each one.

It is also not a ranking signal. Nothing in the spec promises search visibility, and you should be skeptical of anyone who says otherwise. What it does is make your site cheaper and more reliable for an agent to use, which matters more every month as more questions get answered by an assistant reading your docs instead of a person browsing them. We covered the wider picture in answer engine optimization for developer docs.

Here is how an agent actually uses it:

flowchart TD
  Q["User asks an assistant about your product"] --> F["Agent fetches /llms.txt"]
  F --> S["Reads the H1 and summary blockquote"]
  S --> P["Picks the links whose notes match the question"]
  P --> M["Fetches those pages, preferring a Markdown version"]
  M --> A["Answers from the pages it read"]
  P -. "stale or wrong link" .-> X["Answers from the wrong page, confidently"]

Two details in that flow do most of the work. The summary blockquote tells the agent whether it is even on the right site. The link notes are how it decides which of twenty pages to open, so a note that says "API reference" is worth far less than one that says "every endpoint, auth scheme, and error code."

The v2 format, with a worked example#

The spec defines a fixed order. Only the first element is required:

1. An H1 with the project or site name. This is the only mandatory part. # Acme is a valid llms.txt.

2. A blockquote summary. One or two sentences with the key facts an agent needs to understand everything below it.

3. Optional detail sections. Paragraphs or lists, but no headings, for things like "this API is not compatible with X" or "all examples assume version 3." This is where you prevent the most common wrong answers.

4. H2 sections containing file lists. Each list item is a Markdown link, optionally followed by a colon and a note: - [name](url): note.

5. An ## Optional section, by convention. Links an agent can skip when it is short on context. In v2 this is a convention rather than a mechanical rule, but it is still the clearest way to say "secondary."

Version 2 also added three things worth knowing:

• Subpath files. A file at /docs/llms.txt covers pages under /docs/, and when more than one file applies, agents should use the most specific one. That lets a GitHub Pages project site or a docs subfolder participate without owning the domain root.
• Link relations for discovery. Pages can point to their llms.txt with rel="describedby" and to a Markdown version of themselves with rel="alternate" type="text/markdown", as an HTML <link> element or an HTTP Link: header.
• Both Markdown URL styles. A clean Markdown copy of a page can live at page.html.md or with the extension replaced by .md. The spec accepts both.

A worked example#

Here is a complete file for a fictional transactional email API. It is short on purpose. Twenty well-described links beat two hundred undescribed ones.

# Acme Mail

> Acme Mail is a REST API for sending transactional email. All endpoints are under https://api.acme.dev/v2 and authenticate with a bearer API key.

- The v1 API is deprecated and will be shut off in 2027. Never suggest v1 endpoints.
- SDKs exist for Node, Python, and Go. There is no official Ruby SDK.

## Docs

- [Quickstart](https://acme.dev/docs/quickstart.md): Send your first email in five minutes with curl or the Node SDK.
- [Authentication](https://acme.dev/docs/auth.md): API keys, key scopes, and rotating a leaked key.
- [API reference](https://acme.dev/docs/api.md): Every endpoint, request field, and error code.
- [Webhooks](https://acme.dev/docs/webhooks.md): Delivery, bounce, and complaint events, plus signature verification.

## Guides

- [Sending at volume](https://acme.dev/docs/guides/volume.md): Rate limits, batching, and warming a new domain.
- [Domain setup](https://acme.dev/docs/guides/domains.md): SPF, DKIM, and DMARC records for a sending domain.

## Optional

- [Changelog](https://acme.dev/changelog.md): Release history.
- [Status](https://status.acme.dev): Live uptime.

Notice what the detail section does: it states the two facts an assistant is most likely to get wrong (the deprecated version, the missing SDK) as plain declarative bullets. That section is the cheapest accuracy win in the whole file.

If you would rather not hand-write it, the free llms.txt generator imports your sitemap, pulls page titles and meta descriptions into the notes, and gives you a live preview to prune from.

Common mistakes a validator catches#

These are the structural problems that come up most often, and each one is cheap to fix:

The last row catches more sites than you would expect. Many single-page apps return their HTML shell with a 200 status for any path, so curl -I https://yoursite.com/llms.txt looks healthy while the body is a <!doctype html> page.

You can check all of this in a few seconds with the free llms.txt validator. Paste a file, or enter a domain to check the live file. A live check gives you a shareable report page and a status badge for your README. If you want to see how other developer-tool sites are doing, the State of AI-Ready Docs report tracks adoption and spec compliance across a public list of sites.

Serving it so agents find it#

Getting the file right is half the job; the other half is making sure an agent can actually retrieve it.

1. Put it at the root, or at the most specific path you control. /llms.txt covers the whole site. If your docs live on a subpath you own separately, add /docs/llms.txt as well.

2. Return real text. A 200 status, a text/plain or text/markdown content type, and a body that starts with #. Test it with curl rather than a browser, which will happily render an HTML fallback.

curl -sI https://yoursite.com/llms.txt | grep -iE "^(HTTP|content-type)"
curl -s https://yoursite.com/llms.txt | head -5

3. Add the v2 link relations. A header on your HTML pages lets an agent that landed on a deep page find the map without guessing:

Link: </llms.txt>; rel="describedby", </docs/quickstart.md>; rel="alternate"; type="text/markdown"

4. Publish Markdown versions of the pages you link. The spec recommends linking to clean Markdown rather than HTML. An agent reading your Markdown quickstart spends its context on your content, not your navigation.

5. Keep it small enough to fit in context. The spec's own guidance is that the file should stay small enough to fit in a model's context window. If yours runs to hundreds of links, split it with subpath files instead.

The trap: a valid file that is no longer true#

Every check above is structural. A validator can tell you the H1 exists and every link is well-formed. It cannot tell you that the "Authentication" page now describes an OAuth flow you retired in June, or that the note on your API reference still says "v2" after you shipped v3.

That is the real failure mode of llms.txt, and it is the same one every other doc has. The file points at pages; the pages describe code; the code changes every week. An agent that trusts a stale llms.txt does not fail loudly — it confidently quotes the old page, and the user who asked has no reason to doubt it. We have written about why this happens in what is documentation drift, and about putting a number on it in documentation debt.

The practical defense is to treat llms.txt as a view over your docs, not a document in its own right. When a page it links to changes meaningfully, the note should change too. When a page is deleted, the link should go. And the pages themselves need to track the code, or the map is just a clean index to wrong answers.

Keeping the pages behind it accurate#

This is the part Moxie Docs is built for. It indexes your GitHub repository into living, source-cited documentation, flags where docs have fallen behind the code, and opens reviewable pull requests to bring them back in line — it never merges them for you. Coding agents get the same context over the Moxie MCP server, so the agent writing a change sees the conventions and the docs it might break before it commits. A well-formed llms.txt pointing at pages that stay true is the combination that actually earns you accurate answers; you can try it on one repository with the free plan, no card required.

Measuring whether it's working#

• The live file validates. Run the validator against your domain after every docs deploy, or embed the badge so a regression is visible in the README.
• The file returns text, not HTML. A one-line curl in CI catches the SPA-fallback problem before an agent does.
• Every link resolves. A link checker over the URLs in llms.txt takes seconds and catches deleted pages.
• Assistants answer your top questions correctly. Ask two or three assistants the five questions your support team hears most, and check which pages they cite.
• The notes match the pages. Once a quarter, read each note next to the page it describes. If you would not say it to a new hire, rewrite it.

None of these need a dashboard.

Frequently asked questions#

What is llms.txt used for?#

llms.txt gives AI agents and assistants a curated map of a website: a short summary of what the site is, notes that prevent common mistakes, and links to the most important pages with a description of each. Agents read it to decide which pages to open when answering a question, instead of guessing from navigation menus and marketing copy.

Is llms.txt required, and does it help SEO?#

No. It is an optional, community-proposed format, and nothing in the spec promises search rankings. Its value is practical: it makes your site cheaper and more reliable for AI agents to read, which improves the answers assistants give about your product. Treat it as documentation for a new kind of reader, not as an SEO tactic.

What is the difference between llms.txt and llms-full.txt?#

llms.txt is a map: a summary plus links. llms-full.txt is a convention many sites follow of publishing the full text of their docs in one file, so an agent can load everything in a single fetch. The v2 spec itself does not define llms-full.txt, so treat it as optional and keep llms.txt as the primary, curated entry point.

How do I check if my llms.txt is valid?#

Paste the file into an llms.txt validator, or check your live domain so you catch serving problems as well as syntax. A good check confirms the H1 title, the summary blockquote, well-formed - [name](url): note links, absolute URLs, no duplicates, the Optional section placed last, and that the URL returns text rather than an HTML page.

Where should llms.txt be placed?#

At the root of your site, served at https://yoursite.com/llms.txt. Since v2, you can also place one under a subpath, such as /docs/llms.txt, to cover only the pages under it; when more than one applies, agents should use the most specific file.

The short version#

An llms.txt file is a short, curated Markdown map that tells AI agents what your site is and which pages to read. Write the H1 and summary, state the facts assistants most often get wrong, link to clean Markdown pages with notes that say what each page is for, and serve it as real text at the root. Then accept that a valid file is not the same as a true one: the map is only as accurate as the pages it points to, and those pages are only as accurate as your last merge.