llms.txt, Explained
A proposed standard for telling AI assistants what your site is about. Adoption is not universal and the benefit is unproven — but the cost is an afternoon, and the discipline it forces is valuable on its own.
llms.txt is a proposed convention: a plain-text file at your site root summarising what your site is about and linking to your key pages, written for AI assistants rather than browsers. It does not control crawling — that is robots.txt — and support is not universal.
What It Is and What It Isn't
llms.txt is a markdown file at yoursite.com/llms.txt containing a concise, structured summary of your site: what the organisation is, what it offers, and links to the pages that matter, with a sentence explaining each.
The reasoning is straightforward. An assistant trying to understand your site otherwise has to crawl and interpret pages built for humans, complete with navigation, marketing language and layout. A single file stating the same facts plainly removes the guesswork and the opportunity for misinterpretation.
It does not control access. That is robots.txt, which is a long-established standard that AI crawlers do read. llms.txt is descriptive, not restrictive — it says what you are, not who may fetch it. Conflating the two is the most common misunderstanding.
It is also not a ranking mechanism. There is no evidence it affects search rankings, and anyone selling it as an SEO tactic is overselling. Its value, if it has one, is accuracy of description.
What Goes In It
The convention is loose, which is part of why adoption varies. This structure is what most published examples follow and what makes the file genuinely useful regardless of who reads it.
A one-paragraph summary of the organisation
Who you are, what you do, where you operate, who you serve. Written to be quoted directly, because that is the most likely use.
Key pages with a sentence each
Links to your main sections with a line describing what each contains. This is the part that saves an assistant from inferring your structure from a navigation menu.
Facts stated plainly
Contact details, hours, founding date, terms, anything factual that people ask about. Concrete, checkable statements rather than positioning language.
Disambiguation, if your name is ambiguous
If similarly-named companies exist, say explicitly that you are not them. This is genuinely useful — name collisions are a common cause of assistants confidently describing the wrong business.
Optionally, an llms-full.txt
A longer companion with the complete detail — full FAQs, all terms, everything — for cases where the summary is not enough. Link to it from the short file.
Is It Worth Doing?
Honestly: probably, but not for the reason it is usually recommended. The direct benefit is unproven and support is inconsistent. The indirect benefit is real and available immediately.
The direct case is weak at present. Not every assistant fetches the file, none of them publicly commit to weighting it, and there is no way to measure whether yours was used. Anyone claiming a demonstrated ranking or citation benefit is ahead of the evidence.
The indirect case is stronger. Writing the file forces you to state, in plain language, what your business is and what is true about it. Most organisations discover during this exercise that their website never actually says several of these things — and that is a finding worth having regardless of whether any machine reads the result.
There is also a consistency benefit. Once the file exists it becomes the canonical statement of your facts, which makes contradictions elsewhere easier to spot. A price list in this file that your site no longer publishes, for instance, is a discrepancy you want to find before an assistant quotes it back to a customer.
The cost is an afternoon and no ongoing obligation beyond keeping it accurate. At that price the asymmetry favours doing it.
Common Questions
What people ask before adding one.
Does llms.txt replace robots.txt?
No, and they do different jobs. robots.txt controls which crawlers may fetch what, and AI crawlers do respect it. llms.txt describes your content. You need robots.txt regardless; llms.txt is optional and additive.
Will it stop AI training on my content?
No. It has no restrictive function at all. Limiting training use means blocking specific crawlers in robots.txt — and note that blocking them also removes you from the answers those systems produce.
How long should it be?
The short file should be scannable — roughly one to two pages. Put the exhaustive detail in a linked llms-full.txt. A single enormous file defeats the purpose of having a summary.
How often should I update it?
Whenever the facts change, and it should be checked whenever the site changes materially. A stale llms.txt is worse than none, because it states outdated facts with the authority of an official file.