llms.txt: What It Is and Whether It Helps AI Visibility

llms.txt is a proposal for a plain-text file at the root of a website, written in Markdown, that tells AI tools what the site is about and where its most useful pages are. Jeremy Howard of Answer.AI published the proposal on 3 September 2024. Despite the name, it is nothing like robots.txt: it controls nothing and blocks nothing. It is a map, not a rule.

This guide covers what the file contains, what the large 2026 studies found, and what to do first if your goal is to be cited in AI answers.

What goes in the file

The proposal at llmstxt.org describes a Markdown file served at /llms.txt, in this order:

1. An H1 with the name of the site or project. This is the only required part.
2. A short summary in a blockquote.
3. Optional paragraphs or lists with more detail, without headings.
4. Optional sections, each under an H2, listing links as [name](url) with a short note. A section called "Optional" holds links an AI tool can skip when it needs less context.

The reasoning is practical. Web pages are built for people, with navigation, scripts and boilerplate, and a language model works within a limited context window. A short, clean list of your best pages is easier for a tool to use than your whole site.

The proposal calls itself exactly that, a proposal open for community input. It is not a ratified standard.

What the evidence says, as of October 2026

Three sources are worth knowing.

Ahrefs: most files are never read. Ahrefs looked at 137,210 domains in its web analytics product (published June 2026). About 28 percent published an llms.txt, but 97 percent of those files received no requests at all in May 2026. Among the few files that were fetched, 19.5 percent of requests came from named AI tools, and no AI bot requested an llms.txt that did not exist. Ahrefs concluded that, for AI search visibility, the cons outweigh the pros for now, while noting that agents such as coding assistants may still find the file useful.

SE Ranking: no link to citations. An analysis of about 300,000 domains, reported by Search Engine Journal in November 2025, found no relationship between having an llms.txt and how often a domain was cited in AI answers. Removing the llms.txt signal from its model made the model slightly more accurate.

Google: not needed for Search. Google's guide to optimising for generative AI search includes a section on what you do not need to do, and lists llms.txt files among them. It says Google Search does not use such files, and that creating them for other services will neither harm nor help your visibility in Google Search. Separately, Chrome's Lighthouse has an llms.txt audit, but it only flags a server error when the file is requested. A missing file is marked not applicable, because the file is optional.

A fair summary: llms.txt is cheap to add, and nothing measured so far shows it changes how often you are cited. Treat it as an afternoon task, not a strategy. Your own server logs are the only reliable answer for your site.

The file that does matter today: robots.txt

robots.txt controls documented behaviour, and AI companies publish which user agents they use. OpenAI's crawler documentation is the clearest example:

• OAI-SearchBot is for search. Sites that opt out are not shown in ChatGPT search answers, though they can still appear as navigational links.
• GPTBot crawls content that may be used to train OpenAI's models. Blocking it is a training choice, not a search choice.
• ChatGPT-User visits a page when a person asks ChatGPT something that needs it. OpenAI says robots.txt rules may not apply, because a person started the request.

OpenAI says each setting is independent, so you can allow OAI-SearchBot while blocking GPTBot, and that search changes can take about 24 hours to apply. Perplexity documents PerplexityBot for its search results, and Google documents Google-Extended as a token for Gemini training and grounding that does not affect inclusion in Google Search.

The free AI crawler checker reads your robots.txt for 14 AI and search crawlers, and also tells you whether you have an llms.txt.

What to do first

1. Open the doors. Allow the search crawlers of the assistants you care about. How to get cited by ChatGPT goes through the ChatGPT side in detail.
2. Make key pages answer-friendly. Clear claims early, with specifics: the kind of content the GEO research found AI answers lean on.
3. Be present where assistants already look. Comparison pages, reviews, forums and documentation get cited whatever your llms.txt says; see how Perplexity chooses sources.
4. Measure what assistants say about you, then judge llms.txt by your own data: add it, and watch whether anything changes.

If you add one, do it properly

• Keep it short: a sentence or two about the site, then your most important pages with an honest one-line note each.
• Link only pages that answer questions: product facts, pricing, comparisons and documentation.
• Regenerate it when the site changes. A stale map is worse than none.

For an example, Vouch publishes its own at vouchgeo.com/llms.txt, generated from the same list of pages as the sitemap so it cannot drift. The free llms.txt generator and validator builds a file from your pages and checks a live one against the proposal.

Common questions

Is llms.txt a standard? No. It is a 2024 proposal published at llmstxt.org, open for community input. Google says its Search does not use the file, and the large studies above found most files are never requested and no link to citations.

Can llms.txt hurt my SEO? Google says it will neither harm nor help visibility in Google Search. The real cost is the time spent on it instead of work with evidence behind it.

How do I know if AI tools read my llms.txt? Check your server or CDN logs for requests to /llms.txt and look at the user agents. In the Ahrefs sample, 97 percent of files got no requests in a month, so an empty log would be normal, not a sign that you did something wrong.

Look at the answers, not just the files

Whether or not llms.txt pays off, the question underneath is the same: what do AI assistants say when buyers ask about your category? Vouch follows that whole conversation and shows where your brand drops out. Our methodology explains how.

Vouch is invite-only for now, and every invitation starts with a 14-day free trial: join the waitlist.