llms.txt is a plain text file, written in Markdown, that sits at your site’s root and points AI tools to your most important pages. Jeremy Howard of Answer.AI proposed it in 2024. No major AI provider has confirmed that it reads the file, so most sites do not need one yet.
Ask ChatGPT about your own business, and you may not like the answer. It might name the wrong services, quote pricing you retired last year, or skip your best pages completely. That gap is why llms.txt keeps coming up in technical SEO conversations. Here is what the file does, what the data says, and whether your site needs one.
What is an llms.txt file?
An llms.txt file is a curated list of your best content, written for machines rather than people. You save it as Markdown, name it exactly as loads at yourdomain.com/llms.txt, and host it at your root.
The reasoning behind it holds up well. Large language models do not index your whole site the way Google does. They pull pages in real time, work inside limited context windows, and struggle with anything buried under heavy JavaScript. The official llms.txt
specification argues that models benefit from concise, expert-level information gathered in one accessible location, and that converting cluttered HTML into clean text is both difficult and imprecise.
| # Site title > A brief description of the documentation site.## Docs – [API](https://example.com/docs/api.md): Endpoint list and usage- [Install](https://example.com/docs/install.md): Setup steps- [Getting started](https://example.com/docs/start.md): Intro guide ## OpenAPI Specs – [openapi](https://example.com/docs/openapi.json) ## AsyncAPI Specs – [asyncapi](https://example.com/docs/asyncapi.yaml) |
So the file is meant as a shortcut. Rather than hoping a model stumbles onto your pricing page, you hand it a map of what matters.
What does an llms.txt file actually look like?
Here is a short llms.txt example following the structure the specification describes. An H1 for the site name, a blockquote summary, then H2 sections holding linked lists with a short description after each link. That is the entire format, and it is deliberately simple so that parsers and regular expressions can read it reliably.
Who created llms.txt and why?
Jeremy Howard of Answer.AI put the proposal forward in September 2024. He is a well-known figure in machine learning, and the problem he identified is real enough. Models burn tokens untangling navigation menus, cookie banners, and ad slots before they reach anything useful.
Notice the word proposal, though. llms.txt is a suggested convention rather than a ratified standard, and nobody is obliged to honour it. That single distinction explains most of what follows in this article.
How is llms.txt different from robots.txt and sitemap.xml?
These three files sit in the same folder and do completely different jobs. The llms.txt vs robots.txt confusion comes up constantly, so here is a plain comparison.
| File | What it does | Who it speaks to | Format | Is it enforced |
| robots.txt | Controls which crawlers may request which parts of | Search and AI crawlers | Plain text | Yes, honoured by all major crawlers since |
| your site | 1994 | |||
| sitemap.xml | Lists every URL you want | Search | XML | Yes, actively fetched by |
| indexed | engines | Google and Bing | ||
| llms.txt | Suggests which pages matter most and | Largelanguage | Markdown | No, voluntary and unconfirmed |
| summarises them | models |
Robots.txt is the oldest of the three, introduced in 1994 as the Robots Exclusion Protocol. Thirty years of adoption is why it still carries real weight, and it is worth remembering that llms.txt is roughly two years old by comparison.
The llms.txt vs sitemap.xml distinction matters more than most articles admit. Your sitemap gets discovered automatically, because search engines look for it at a known path or find it referenced inside robots.txt, then use it to guide crawling and manage crawl budget on larger sites. Nothing pulls llms.txt in the same way. It has to be requested deliberately. All three can live in your root directory together. Adding one never replaces another.
Do ChatGPT, Claude, and Gemini actually read llms.txt?
No, not in any confirmed way. As of mid-2026, no major provider has formally adopted llms.txt as part of its crawler behaviour, and the server log evidence points in the same direction. This is the part worth reading twice, because it contradicts most published advice on the subject.
What the adoption data shows
Ahrefs ran the largest study available on this. They analysed 137,000 domains and found that 28 percent published a valid llms.txt file, which works out to 38,360 sites. More than one in four. On adoption alone, the file looks like it is winning.
Then they checked whether anything read those files. Of the roughly 38,000 domains with a valid file, 97 percent received zero requests for it during May 2026. No bots, no humans, nothing at all. Semrush found the same pattern at a smaller scale. They added the file to Search Engine Land in March 2025, then reviewed the server logs. Between mid August and late October 2025, the file received zero visits from Google-Extended, GPTBot, PerplexityBot, or ClaudeBot. Googlebot and Bingbot touched it a handful of times and treated it like any other page on the site.
Two adoption figures worth reconciling, since you will see both quoted. Semrush cited NerdyData showing only 951 domains with an llms.txt file as of July 2025, which sits nowhere near 28 percent. The samples are not comparable. Ahrefs measured sites already running their analytics software, meaning technically minded owners who follow AI news closely, while the NerdyData count came from a broader crawl a full year earlier. Read 28 percent as an upper bound among engaged site owners rather than a figure for the web at large.
Which AI companies support llms.txt?
None of them formally, though the picture is not uniform.
OpenAI honours robots.txt and runs separate agents for different jobs, with OAI-SearchBot handling ChatGPT search and GPTBot handling training. Neither is documented as reading llms.txt. Anthropic publishes its own llms.txt file for its documentation, which suggests openness to the idea, but has not said ClaudeBot consumes them. Google manages AI crawl behaviour through the Google-Extended token in robots.txt, with no mention of llms.txt support. Meta has published no crawler guidance on it at all.
Google did include llms.txt inside its Agent2Agent protocol in April 2025, which Ryan Law of Ahrefs fairly described as adding one proposed protocol to another. No commitment to crawl it came with that.
Why does Google keep sending mixed signals?
Because two different teams inside Google have said opposite things within the same week. In late May 2026, its guide on optimising for generative AI features included a mythbusting section stating plainly that machine-readable files like llms.txt are not needed to appear in AI Overviews or AI Mode. Days later, the Chrome team shipped an llms.txt check inside.
Lighthouse’s experimental agentic browsing audits describe it as an emerging convention that gives agents a machine-readable summary of a site.
John Mueller of Google explained the gap when pressed on it. His position is that llms.txt is not done for search, and works more as a temporary crutch that saves tokens for AI coding tools reading developer documentation. He has also compared it to the old keywords meta tag, because it is a claim a site owner makes about their own site rather than anything verified.
That comparison drew pushback, and fairly so. Carolyn Shelby of Yoast argued in Search Engine Land that the analogy undersells it, since meta keywords made unverifiable claims while llms.txt points to real URLs that either deliver or do not. Both things can be true at once. The file is harder to game than meta keywords ever was, and almost nothing is reading it.
What is llms-full.txt and how is it different?
llms-full.txt is the expanded version, containing the actual text of your pages rather than a list of links. Picture llms.txt as a table of contents and llms-full.txt as the whole book flattened into a single file.
File sizes climb fast. One site owner writing in Search Engine Land described their own version running to 115,378 words and 966 kilobytes. Everything above about adoption applies equally here. If almost nobody requests the short file, the long one is not being read either.
Does your website need an llms.txt file in 2026?
Probably not, and that is our honest read at Affordable SEO Services. The realistic answer depends entirely on what kind of site you run, so here is how we split it.
When adding one makes sense
You have a genuine case if you publish developer documentation, an API reference, or a technical knowledge base. Mueller’s own framing supports exactly this use, since AI coding assistants parse documentation constantly and a clean Markdown index saves them real work.
Cost matters too. Yoast SEO now ships a one-click llms.txt feature that generates and refreshes the file automatically. If your CMS handles it and you never think about it again, there is no sensible reason to switch it off.
Look at who actually publishes one. Anthropic, Cloudflare, Mintlify, Hugging Face, Vercel, Zapier, and Cal.com are the names that come up repeatedly, and every one of them is documentation-heavy. Nobody has yet shown a plumber in Birmingham getting more calls because of a text file.
Does a small business or local site need one?
No, in almost every case. If you run a local service business, an ecommerce store, or a marketing site with twenty or thirty pages, models can already read your content provided the HTML is clean. You would be solving a problem you do not have.
Skip it as well if creating one means paying a developer. A few hundred dollars on a file that goes unrequested 97 percent of the time is money that would work harder almost anywhere else on your site. And skip it if you would build it once then forget about it, which brings us to the part most articles leave out entirely.
What are the risks of publishing an llms.txt file?
There is no ranking penalty for publishing one. That does not make it free.
You are handing competitors a map of your best content.
Patrick Stox of Ahrefs raised this publicly, and it is worth sitting with for a moment. A well-made llms.txt is a curated index of your highest value pages, complete with descriptions, in a format built specifically for machine parsing.
Scrapers find that useful. So do competitors doing content research, who now have your content audit already done for them at a predictable URL. This is an argument rather than a measured finding, and reasonable people weigh it differently. For most sites the risk is small, but if your competitive edge is a large content library, it deserves a moment’s thought before you publish.
A stale file is worse than no file at all.
A file listing pages you deleted eight months ago tells any system reading it that your information is unreliable. Broken links, retired services, pricing that no longer applies. Should anything start consuming these files at scale, a neglected one turns into a liability rather than an asset, and you would have no way of knowing. Automate it or do not publish it.
How to create an llms.txt file step by step
Decided it fits your site? Here is how to create llms.txt without a developer.
1. List your priority pages. Choose services, pricing, main guides, and contact, usually fifteen to thirty URLs in total.
2. Open a plain text editor. Notepad, TextEdit in plain text mode, or VS Code all work fine.
3. Write the Markdown. Start with an H1 site name, add a blockquote summary, then group links under H2 headings with a short description after each.
4. Save the file as llms.txt. The name is case-sensitive on most servers, so llm.txt or LLMs.txt will not be found.
5. Upload it to your root directory. In cPanel, that means the public_html folder, so the file resolves at yourdomain.com/llms.txt.
6. Confirm it returns a 200 status. Open the URL in a browser and check the server sends it as text/plain rather than downloading it.
7. Diarise a quarterly review. Strip out dead URLs and add anything significant you have published since.
Would rather not write it by hand? An llms.txt generator such as the free tool from Firecrawl will build a draft from your existing sitemap, which you then trim down to the pages that genuinely matter.
What are the llms.txt best practices worth following?
Keep the file short and curated, because a list of every URL you own defeats the purpose. Write real descriptions rather than repeating page titles, since the description is the only context a model gets. Save as UTF-8, serve over HTTPS, purge your CDN cache after any edit, and never list pages that are noindexed or blocked in robots.txt.
How to add llms.txt to WordPress
Yoast SEO is the quickest route. Go to Yoast SEO, then Settings, then Site Features, find the AI discovery file card, toggle it on, and save. Yoast regenerates the file weekly using WordPress cron, pulling in recently updated posts and pages without any input from you. Several dedicated plugins do the same job if you use a different SEO plugin in WordPress.
How to add it on Shopify, Webflow, or a custom build
Shopify and Webflow both restrict what you can write to the root directory, so you will usually need an app, a redirect, or a small edit through your theme files. On a custom build, it is a two-minute job for whoever manages your hosting.
What should you do instead for AI visibility?
Focus on the work that pays off, whether or not llms.txt is ever adopted. These are the items we prioritise for clients at Affordable SEO Services, and all of them improve ordinary search performance at the same time.
Serve your content in HTML that loads without JavaScript. This is the single biggest one, because AI crawlers mostly read raw HTML, and anything rendered client-side may not exist as far as they are concerned. Making your pages AI-readable in this sense is not a separate project from technical SEO; it is technical SEO.
Get your robots.txt rules deliberate rather than default. OAI-SearchBot, GPTBot, and Google-Extended are real, documented, enforced controls, which is more than llms.txt can claim. Decide which you want reading your content and write the rules to match.
Add schema markup to your key pages. Structured data has been supported and documented for a decade, and it does the job people hope llms.txt will do, which is telling machines what your content means rather than just where it sits.
Answer real questions in clear, short blocks. Forty-to sixty-word answers placed directly beneath a question heading get pulled into AI Overviews far more reliably than any root file. That approach sits at the heart of AEO, or answer engine optimisation, and it works today rather than hypothetically.
Earn mentions on sites that AI tools already cite. Generative engine optimisation, usually shortened to GEO, is largely about being referenced by sources these models already trust. A mention in a roundup on an established industry site shapes what ChatGPT says about you far more than anything hosted on your own domain.
Every item on that list improves content discoverability for humans and machines together. That is what makes them worth the hours, and it is the reason we rarely lead a project with a text file.
Where this leaves you
llms.txt is a reasonable idea with no evidence behind it yet. Publish one if your CMS makes it free, if you run documentation, or if you simply want to be early. Do not pay for one, and do not let it pull attention away from clean HTML, schema markup, and content that answers questions directly, because those three things decide whether AI tools can use your site at all. If you would rather have someone check which of these actually apply to your site, Affordable SEO Services can audit your technical setup and AI visibility together, then tell you honestly where your hours are best spent.
Frequently asked questions
What is the llms.txt meaning?
It is a Markdown file at your site’s root that lists and describes your most important pages for large language models.
Is llms.txt the same as robots.txt?
No. Robots.txt controls crawler access and is enforced, while llms.txt only suggests which content matters and is not enforced by anyone.
Does llms.txt help SEO or affect Google rankings?
No. Google has stated that machine-readable files like llms.txt are not needed to appear in its AI features, and it does not influence rankings.
Does llms.txt work in practice?
Not yet in any measurable way. Ahrefs found 97 percent of valid files received no requests at all during May 2026.
Where exactly do I put the llms.txt file?
In your root directory, so it loads at yourdomain.com/llms.txt over HTTPS and returns a 200 status.
Is there a free llms.txt generator?
Yes. Firecrawl offers a free generator, and Yoast SEO creates the file automatically on WordPress.
What happens if I do not have an llms.txt file?
Nothing measurable. No AI system currently requires one, and the overwhelming majority ofpublished files are never requested.
How often should I update llms.txt?
Quarterly at minimum, or automatically through your CMS. An outdated file listing deleted pages does more harm than good.
Does llms.txt work on WordPress?
Yes. Yoast SEO generates and refreshes it from your dashboard with no code required.
