llms.txt: The Markdown File for Agents, and What Reads One
Almost nothing fetches this file yet. Before you write one, see what belongs inside it, who actually publishes one, and whether your site should.
/ On this page7 sections
An llms.txt file is a markdown file at your site's root that lists the pages worth reading, written for software rather than for people.
Almost nothing fetches one yet.
That is not an argument against writing one. It is a reason to know what the file does before you write it.
LLMs.txt Is a Proposal, Not a Standard
llms.txt is one person's proposal. It is not a standard, and no standards body has adopted it.
Jeremy Howard published it at llmstxt.org in September 2024 and revised it in August 2026. Its own summary calls it a proposal "to standardise on using an /llms.txt file to provide information to help agents use a website".
Nothing obliges you to have one.
There is no registry to join, no validator that anybody checks, and no consequence for a site that never publishes one.
The reasoning underneath it is sound, and it is the honest part of the case for publishing one.
A web page wraps its information in navigation, scripts and layout. Software reading that page pays for all of it before reaching a sentence worth having.
So the proposal's answer is a curated index. One short markdown page that says what the site is, then links to the pages an agent should read, with a line of explanation beside each link.
That is the idea in full. Everything after it is format.
Inside the File
Inside, the shape is short and fixed: a title, a summary, then lists of links with a note beside each one.
The parts run in one required order, top to bottom:
- An H1 with the name of the site or project. llmstxt.org calls this "the only required section", so a file carrying nothing else is still a valid file.
- A blockquote summary, one short paragraph, holding the context needed to make sense of what follows.
- Any amount of ordinary markdown with no headings in it, explaining the site and how to read the files it lists.
- Sections marked with an H2, each holding a list of links written as
[name](url), with an optional note after a colon.
By convention a section headed Optional holds links an agent can skip when it needs a shorter context. The August 2026 revision took the mechanical meaning out of that section and left it as a convention.
The example llmstxt.org publishes is a cut-down version of the FastHTML library's file. An H1 naming the library, a blockquote describing it in a sentence, two notes about what it is not compatible with, then link lists headed Docs, Examples and Optional.
Read that shape twice and you have the format.
The work in writing one is choosing which pages belong in the lists. There is almost nothing else to it.
The lists are the work
Put your own site through the four parts, in the order the format asks for them, and watch how little there is to it. Nothing here is sent anywhere, and nothing is saved.
Part one, and the only required one
An H1 with the name of the site or project
llmstxt.org calls this the only required section, so a file carrying nothing else is still a valid file.
Part two
A blockquote summary
One short paragraph, holding the context needed to make sense of what follows. Say in one sentence what the site is and who it is for.
Part three
Any amount of ordinary markdown with no headings in it
Explaining the site and how to read the files it lists. Leave it empty if you have nothing to add.
Part four, and this is the part that takes thought
Sections marked with an H2, each holding a list of links
Pick the pages software should read. Documentation, reference pages, policies, anything a question about your business would be answered from. Not your campaign landing pages.
The note after a link is the part doing the work, because it lets an agent choose which link to follow instead of fetching all of them.
The file, in the order the format requires
# nothing typed yetType a name in the first box and the file above is finished.
Then it is a text file and an upload. Save what you typed as a file called llms.txt, put it at the root of your site, and load it in a browser at your domain followed by /llms.txt before you call it done.
The Markdown Twin, and the Full-Text File
The proposal asks for a second thing alongside the index, and no fetch count covers it. Ahrefs says its study "measures the index file, and only the index file".
llmstxt.org asks sites to serve a clean markdown version of the pages an agent might need, at the same address, with .md appended or substituted, so software following a link is handed prose rather than a page.
Documentation sites have started doing it. Checked in September 2026, Cloudflare's developer docs and GitHub Docs both advertise a markdown version of a page through a rel="alternate" link element.
llms-full.txt is the other file you will meet, and it is not in the specification.
It is a convention documentation platforms settled on: a full-text file holding an entire documentation set at once, rather than an index pointing at pages. Mintlify and Nuxt UI both serve one.
The two are not interchangeable, and the index is the one to start with. An entire documentation set in a single file can run past a standard context window, which is why Nuxt UI publishes both and points readers at the index first.
Almost Nothing Fetches One Yet
Publishing an llms.txt file and having one read are different things, and the distance between them is the answer to whether this is worth your afternoon.
Ahrefs measured every request to these files across 137,210 domains on its own analytics in May 2026. Of those domains, 28% published a file, and 97% of the files it found were fetched by nothing at all that month.
Its own caveat travels with the number. Ahrefs notes that its customers "skew more technical and SEO-aware than the web at large", so treat that adoption share as an upper bound rather than a reading of the whole web.
Nothing goes looking for one.
Requests for llms.txt files that do not exist drew no AI bot at all in the same Ahrefs study. That is the quieter finding, and it matters more than the headline number.
On that evidence a file is fetched when something has already been handed the address. That is a different mechanism from the one people think they are buying.
Even the check built into Chrome's own auditor barely registers. Google's Chrome team added it to Lighthouse in 2026, in an experimental category for proposed standards, and Ahrefs counted those audit requests at roughly 1 in 1,000 of the fetches it saw.
The Readers That Do Exist
Most of what reaches these files is not AI at all.
Ahrefs put SEO audit tools top at 21.7% of requests in May 2026, with the four AI categories together at 19.5%. About half of that audit share is Ahrefs' own crawlers.
Inside that AI slice Ahrefs put GPTBot first and Anthropic's Claude Code second, ahead of every AI search bot and every assistant. One is a training crawler, the other a coding agent.
These files are used most heavily in software documentation, where coding agents follow them to reach API references and tutorials. That is llmstxt.org's own reading of where the proposal took hold.
The mechanism is manual. In a coding tool such as Cursor or Windsurf you point at the file with an @ mention, and it is read in as context.
Somebody hands a URL to an agent, the agent fetches it, and the file earns its place by being short and well organized when it arrives.
The AI labs publish for that reason. Checked in September 2026, Anthropic, the Gemini API documentation, Cloudflare, Stripe, GitHub and Perplexity all serve one from their documentation, and llmstxt.org names OpenAI, Anthropic and Gemini as labs publishing for their own developer docs.
Publishing for your own documentation says nothing about what a large language model (LLM) provider supports on anybody else's site. Ahrefs put that flatly in June 2026: "no major LLM provider currently supports llms.txt. Not OpenAI. Not Anthropic. Not Google."
Use this chart — embed code and citation
<a href="https://neerajjivnani.com/blog/llms-txt/"><img src="https://neerajjivnani.com/infographics/llms-txt/published-and-then-nothing.png" alt="Two bars measuring the gap between publishing an llms.txt file and having one read, from Ahrefs' study of every request to these files across 137,210 domains on its own analytics in May 2026. The first bar, 28%, is the share of those domains that published a file. The second bar, 97%, is the share of the files Ahrefs found that were fetched by nothing at all that month, and both bars are drawn to the same zero to one hundred percent scale against their own stated bases. A highlighted panel underneath says Ahrefs looked at requests for llms.txt files that do not exist and found no AI bot among them at all, so on that evidence a file is fetched when something has already been handed the address. A band at the foot says Ahrefs notes its customers skew more technical and SEO-aware than the web at large, so 28% is a reading of that population rather than of the whole web." width="1200"></a>
<p>Chart: <a href="https://neerajjivnani.com/blog/llms-txt/">Neeraj Jivnani</a></p>Neeraj Jivnani, "llms.txt: The Markdown File for Agents, and What Reads One", neerajjivnani.com, https://neerajjivnani.com/blog/llms-txt/Free to republish with a link back to this page.
Use this chart — embed code and citation
<a href="https://neerajjivnani.com/blog/llms-txt/"><img src="https://neerajjivnani.com/infographics/llms-txt/publishing-is-not-reading.png" alt="Two columns setting who publishes an llms.txt file against who officially supports one. The left column, checked in September 2026, lists Anthropic, the Gemini API documentation, Cloudflare, Stripe, GitHub and Perplexity as serving a file from their own documentation, and notes that llmstxt.org names OpenAI, Anthropic and Gemini as labs publishing for their own developer docs. The right column, highlighted, carries Ahrefs' statement from June 2026 that no major LLM provider currently supports llms.txt, not OpenAI, not Anthropic and not Google, and points out that Anthropic sits on both sides of the figure. A band across the foot says most of what reaches these files is not AI at all, and the AI requests that do arrive come from bots that were handed the address rather than ones that went looking." width="1200"></a>
<p>Chart: <a href="https://neerajjivnani.com/blog/llms-txt/">Neeraj Jivnani</a></p>Neeraj Jivnani, "llms.txt: The Markdown File for Agents, and What Reads One", neerajjivnani.com, https://neerajjivnani.com/blog/llms-txt/Free to republish with a link back to this page.
Google Search Ignores It, and Says So
An llms.txt file will not move your Google rankings.
Google's guide to optimizing for generative AI features, last updated in July 2026, files llms.txt under a heading called "Mythbusting generative AI search: what you don't need to do".
Keeping one, it says, "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them".
Both halves of that sentence matter. Google also calls publishing one for other systems "completely fine", so nothing in its position argues for taking a file down once it exists.
It Is Not Robots.txt, and It Blocks Nothing
Three files can sit at your site's root doing three unrelated jobs. Only robots.txt controls anything.
| The file | What it does | Who it is for |
|---|---|---|
robots.txt | Says which paths a named crawler may request | Crawlers that choose to obey it |
sitemap.xml | Lists the pages you want indexed | Search engines |
llms.txt | Points at the pages worth reading, with notes | Agents that have been given the address |
llmstxt.org draws the line itself. robots.txt "lets automated tools know what access to a site is considered acceptable", while the information in llms.txt "is instead used on demand, when an agent needs information about a topic while assisting a user".
The consequence is the part to remember. An llms.txt file grants nothing and withholds nothing, so it cannot block a model that robots.txt allows, and cannot admit one that robots.txt has turned away.
If access is what you care about, robots.txt is still the file that decides it.
Writing One for Your Own Site
Writing one needs a text editor and an upload.
Work in this order:
- Pick the pages software should read. Documentation, reference pages, policies, anything a question about your business would be answered from. Not your campaign landing pages.
- Write one line about each. The note after a link is the part doing the work, because it lets an agent choose which link to follow instead of fetching all of them.
- Open with the H1 and the blockquote. Name the site, then say in one sentence what it is and who it is for.
- Save it as a text file called
llms.txt, and upload it to the root of your site. The markdown is nothing more than the text you typed. - Load it in a browser at your domain followed by
/llms.txt, before you call it done.
Several platforms will do all of that for you. llmstxt.org lists Mintlify, GitBook, Wix, Yoast SEO and AIOSEO as tools that generate the file automatically.
Automatic generation has one wrinkle, and it matters before you edit anything. Wix's help documentation says that once you edit the file by hand, "it will stop updating automatically so your changes are preserved", and that resetting it to the default version resumes the updates.
Where the File Goes
The root is the default, and it is not the only place a file may sit.
llmstxt.org allows one at any path, covering the pages under that path, and says the most specific file wins where more than one applies. Documentation living at /docs/ can carry its own file at /docs/llms.txt.
That matters when you do not control the root of the host you publish on. A project site sharing a domain can still publish a file for its own directory.
Make One If You Publish Documentation
Write one if your site is developer documentation, an API reference or a product manual.
That is where the readers are, and it is what the proposal was built around. A well organized index helps the person who has pointed an agent at your docs.
For a marketing site, a blog, a catalog or a local business, write one only if you are content for nothing to happen.
It will not move your rankings.
The file does not control how a model describes you, and it does not earn you a citation. It does not put you on any system's radar either, because the systems reading these files do not go hunting for them.
The markdown version of your pages is the more useful thing to publish, and if you are on a documentation platform it is already being done for you.
None of which is a reason to be sour about the idea. It is a young proposal, it was revised in August 2026 to match how agents behave now, and something large may yet adopt it.
Publish one if it suits your site. Publish it knowing what reads it.