Share of answer is the operating metric for this practice.

How we count it
Request an audit

Insight · 10 min

llms.txt is a map, not a ranking signal

llms.txt is a Markdown file served at /llms.txt that gives language models a short summary of a site and a curated list of links to its most useful pages. Jeremy Howard of Answer.AI proposed it on September 3, 2024, and the spec lives at llmstxt.org. As of November 2026 no major AI search engine has said it uses the file for ranking or citation, so treat it as a map for agents, not a visibility lever.

In short

  • The spec requires one thing, an H1 with the site name. A blockquote summary, free-text notes, and H2 sections of links are optional, in a fixed order.
  • Google says Search ignores llms.txt. OpenAI, Anthropic, and Perplexity publish llms.txt files for their own docs, but none of them documents its crawlers reading other sites’ files.
  • Server-log studies agree: most llms.txt files are never requested, and the requests that do arrive come mostly from tools, not AI retrieval bots.
  • It is worth having on documentation sites, where coding agents and IDE tools load docs by URL. On a marketing site it costs an hour and buys very little.
  • Serving Markdown copies to AI bots showed no significant lift in bot visits in the one public controlled test.
  • It is not access control. That is robots.txt.

What is llms.txt?

llms.txt is a proposed convention, not a standard ratified by any standards body. Howard’s post, “/llms.txt—a proposal to provide information to help LLMs use websites,” argued that context windows are too small for whole sites and that HTML full of navigation, ads, and scripts is hard to turn into clean text. The file gives an agent a curated starting point instead.

The proposal compares itself to existing root files. robots.txt says which automated access is acceptable. sitemap.xml lists every indexable page. llms.txt is a short, editorial overview meant to be read on demand, when an agent is helping a user, and the spec expects it to be used mainly at inference rather than for training.

What format does the llms.txt spec require?

The spec defines a Markdown file with sections in a fixed order, and only the first is required.

OrderElementRequiredWhat goes in it
1H1YesThe name of the project or site
2BlockquoteNoA short summary with the key facts needed to understand the rest
3NotesNoParagraphs or lists of any kind, but no headings
4H2 file listsNoLists of Markdown links, each with an optional note after a colon
5## OptionalNoSecondary links an agent can skip when it needs a shorter context

The file sits at the root, /llms.txt, or at any subpath, where it covers the pages beneath that path. When more than one applies, the most specific file wins. That lets a team that controls only /docs on a shared host publish its own.

The spec also asks that pages agents might need offer a clean Markdown version at the same URL with .md appended. URLs without a file name append index.html.md. The current spec page recommends standard link relations so clients can find both files: rel="alternate" type="text/markdown" from a page to its Markdown version, and rel="describedby" from a page to the llms.txt that covers it, either as <link> elements or as an HTTP Link header.

What about llms-full.txt?

llms-full.txt is a convention, not part of the spec. It is a single file containing the full text of the documentation, built for pasting into a context window. Mintlify began generating it automatically for hosted docs sites in November 2024, and Anthropic’s developer docs link to one from their llms.txt. The original Answer.AI proposal describes a related tool, llms_txt2ctx, which expands an llms.txt into llms-ctx.txt (without the Optional links) and llms-ctx-full.txt (with them).

Who actually reads llms.txt?

As of November 2026, the documented readers are developer tools and agents that a user points at a file. The large AI search crawlers have not documented reading it.

PartyWhat is on the record
Google SearchIts guide to generative AI features says Search does not use llms.txt, and that creating one “will neither harm nor help” visibility or rankings. Gary Illyes said in 2025 it was “very easy to draw a parallel” with the 1990s keywords meta tag. John Mueller said in April 2025 that no AI service had said it uses the file.
Google ChromeLighthouse has an llms.txt audit in its agentic browsing category. It flags a server error on fetch and marks a 404 as not applicable, because the file is “optional at the moment.” It checks that the file is retrievable, not that it helps.
OpenAIPublishes llms.txt for its developer docs. Its crawler documentation for GPTBot, OAI-SearchBot, and ChatGPT-User says nothing about reading llms.txt on other sites.
AnthropicPublishes llms.txt and llms-full.txt for its developer docs. Its help article on ClaudeBot, Claude-User, and Claude-SearchBot does not mention llms.txt.
PerplexityPublishes llms.txt for its API docs. Its crawler page for PerplexityBot and Perplexity-User does not mention reading the file on other sites.

Publishing a file for your own docs is not the same as consuming everyone else’s. Several guides cite the first as proof of the second. It is not.

What server logs show

The largest public study, from Ahrefs in June 2026, looked at 137,210 domains in May 2026. 28% published an llms.txt, and 97% of those files received zero requests that month. Named AI bots made up 19.5% of the requests that did arrive, and AI retrieval bots, the kind that fetch pages to answer a question, made up 1.1%. SEO audit tools were the largest single category. Ahrefs notes its customers skew technical, so the 28% figure is an upper bound.

A smaller study from EZY.ai, covering 83 sites over 12 weeks in 2026, counted 3,990 robots.txt fetches from OpenAI’s crawlers against 7 llms.txt fetches, and 3,120 against 9 for ClaudeBot. PerplexityBot fetched robots.txt 775 times and llms.txt never. EZY sells a tool that generates llms.txt, which makes the low numbers more credible, not less.

A fetch is not use. A crawler that downloads the file shows only that it was downloaded. No public study links an llms.txt to a change in what an AI answer cites.

When is llms.txt worth having?

llms.txt is worth having when something will be pointed at it on purpose. That mostly means documentation.

  • Developer docs. Coding agents and IDEs load docs by URL. LangChain’s mcpdoc, for example, is an MCP server that gives Cursor, Windsurf, Claude Desktop, and Claude Code a list of llms.txt files and a tool to fetch the links inside them.
  • Products people ask agents about. A user who pastes your /llms.txt into a chat gets a cleaner start than one who pastes your homepage.
  • Sites with a canonical-page problem. Writing the file forces a decision about which URL is the source for each topic. That decision is useful even if no bot ever reads the result.

It does not do the following:

  • It is not a ranking factor in Google Search, and no AI engine has said it is a citation factor.
  • It is not access control. Nothing in the spec blocks a crawler or grants permission. Use robots.txt for that, and see robots.txt for AI crawlers for the user agents.
  • It does not fix the pages it points at. A thin page listed in llms.txt is still a thin page. What gets cited covers what makes a claim reusable.

Should you serve Markdown versions of pages to bots?

Not for visibility, on the evidence so far. The one controlled test of Markdown serving found no significant lift in bot traffic, and Google says Search does not need it. Markdown copies are worth it where the llms.txt file is worth it: documentation that agents load on purpose.

There are two ways to do it, and they carry different risks.

MethodHow it worksRisk
Parallel .md URLsPublish page.md beside page, as the spec proposes, and point to it with rel="alternate" type="text/markdown"Low. Humans and bots can reach both. The HTML stays canonical
Format swapping by user agentDetect the bot and return Markdown at the same URL, while people get HTMLHigher. Google defines cloaking as “presenting different content to users and search engines with the intent to manipulate search rankings and mislead users.” The same text in another format is not that, but never swap formats for Googlebot, and keep the content identical

Profound ran the test. On 381 pages across six sites, it served Markdown to AI bots on 192 pages and normal HTML on 189, and tracked bot visits from January 19 to February 8, 2026. The Markdown pages got about one extra visit at the median, which Profound reported as “consistent with random noise.” The experiment could detect effects above 40%, so a smaller effect remains possible. ChatGPT-User made up about 73% of the bot traffic. Profound sells AI visibility tools and published a null result about a tactic its market sells, which counts in its favor. It says it plans to re-run the test later in 2026.

SE Ranking looked at the same question from the llms.txt side. Its analysis of nearly 300,000 domains, first reported in November 2025, found an llms.txt file on 10.13% of them and no measurable relationship between having one and how often a domain was cited in sampled LLM answers. Removing the llms.txt variable from its XGBoost model slightly improved the model’s accuracy. SE Ranking does not name the engines it sampled or report coefficients, so read it as a strong absence of evidence, not proof of zero effect.

Google’s position is explicit. Its guide to generative AI features says you “don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search,” because Search “doesn’t use them.”

The useful test for a Markdown copy is the same as for llms.txt: will something be pointed at it on purpose? A coding agent loading your API reference, yes. An answer engine choosing which vendor to cite, not on any evidence published so far.

How do you write an llms.txt file?

Writing one takes an hour for a small site. The work is choosing the links, not formatting them.

  1. List the canonical pages. Pick the URLs you would want quoted: product or service pages, method notes, pricing, documentation entry points, original data. Leave out tag pages, archives, and duplicates.
  2. Write the H1 and blockquote. The H1 is the site or project name. The blockquote is one or two sentences stating what the organization does, in the words you would want repeated.
  3. Add notes only if they prevent a mistake. The FastHTML example in the spec uses this space to say the library is not compatible with FastAPI syntax. That is the right use: a fact an agent would otherwise get wrong.
  4. Group the links under H2 headings. Use plain section names. Each entry is - [Title](URL): note. The note says what the page contains, not why it is great.
  5. Move secondary links to ## Optional. Anything an agent could skip on a short context goes there.
  6. Publish Markdown versions if you can. For docs, serve page.md beside page.html, and link to the .md URLs.
  7. Serve it correctly. Put it at /llms.txt, return 200 as plain text or Markdown, and make sure robots.txt does not block it.
  8. Regenerate it on publish. Tie the file to the build or the CMS so a new page adds a line and a deleted page removes one.

A worked example: the AIEOLABS file

This is the file this site served until this essay was written.

# AIEOLABS

> Professional services for SEO, answer engine optimization, and generative engine optimization. One query universe, scored on classic search, extracted answers, and generative citations.

AIEOLABS is a measurement-led practice. Rankings, extracted passages, and generative citations are reported as separate numbers. Illustrative readouts on the site are labeled as such and are not client results.

## Practice

- [Platform](https://aieolabs.com/platform): The measurement system: query universe, entity model, evidence library, technical integrity, citation watch, executive readout.
- [SEO](https://aieolabs.com/seo): Retrieval. Crawl, architecture, templates, and the right ranking URL.
- [AEO](https://aieolabs.com/aeo): Extraction. Passages, evidence, and structured data for snippets and AI Overviews.
- [GEO](https://aieolabs.com/geo): Citation. Mentions, citations, and links in ChatGPT, Perplexity, Gemini, Claude, and Copilot.
- [Engagement](https://aieolabs.com/engagement): Diagnostic (4 weeks), quarterly program, or an embedded year.
- [Work](https://aieolabs.com/work): Sample readouts for fictional companies, showing the shape of a diagnostic.

## Notes

- [SEO, AEO, and GEO are one system](https://aieolabs.com/insights/one-system)
- [What gets cited](https://aieolabs.com/insights/what-gets-cited)
- [Share of answer](https://aieolabs.com/insights/share-of-answer)

## Contact

- [Request an audit](https://aieolabs.com/contact)
- Email: hello@aieolabs.com

What it does right:

  • The blockquote states what the firm does in one sentence.
  • The notes paragraph exists to prevent a specific error: an agent presenting the sample readouts as client results. That is the FastHTML pattern applied to a services firm.
  • Every Practice link carries a note that says what the page holds.

What it gets wrong, by the spec’s standard:

  • Email: hello@aieolabs.com is not a link. The spec defines file-list entries as Markdown links, so that line falls outside the format a parser expects. [Email](mailto:hello@aieolabs.com) fixes it.
  • The Notes entries have no descriptions, so an agent has to fetch each one to learn what it covers.
  • There is no ## Optional section. Work, the sample readouts, belongs there: useful context, never the page to quote.
  • It lists three essays. New essays, this one included, need a line, or the map goes stale the day they publish.
  • There are no .md versions of the pages. For a services site that is a reasonable omission. For a docs site it would be the main gap.

We fixed the first four the same day. The current file renames Notes to Insights and gives every essay a one-line note, makes the email a mailto: link, and moves Work and Privacy under ## Optional with an instruction not to quote the samples as outcomes. The .md versions stay out of scope for now.

Common mistakes

  • Treating it as an SEO tactic. Google has said in writing that Search ignores it. Report it as agent support, not as a visibility metric.
  • Using it to block crawlers. Disallow rules in llms.txt do nothing. Access lives in robots.txt.
  • Dumping the sitemap. A file with every URL defeats the purpose. The spec calls for a curated overview small enough to fit in context.
  • Marketing copy in the blockquote. An agent will repeat what you write. Write the sentence you would accept in an answer.
  • Pointing at the wrong URL. Link the canonical, indexable version. A link to a redirect, a parameter variant, or a PDF with no HTML twin sends the agent somewhere you did not choose.
  • Letting it go stale. A hand-written file drifts within a quarter. Generate it, or put it on the publishing checklist.
  • Serving errors. A server error on /llms.txt fails the Lighthouse audit. A 404 does not.

Sources

Bring us the domain.

We will tell you where you are absent, which page is in the way, and what has to be true before a citation is possible.

Request an audit