llms.txt: does it actually work? What the server logs and 300,000-domain studies say

What the llms.txt specification actually requires

llms.txt is a single Markdown file at your domain root that offers a large language model a curated map of your site, and its specification asks for far less than most people assume. Jeremy Howard of Answer.AI proposed it on 3 September 2024, and the canonical specification lives at llmstxt.org. The whole thing is a good deal smaller than the volume of argument that has grown up around it over the past two years.

The specification defines exactly one required element: an H1 heading with the name of the site or project. That is it. As the spec puts it, that H1 "is the only required section." Everything else is optional: a blockquote summary of what the site is, then H2-delimited sections listing links with short descriptions. You could satisfy the specification with a single line of Markdown. That minimalism is deliberate, and it is also why the file is cheap to ship and easy to get wrong.

llms.txt is the map, llms-full.txt is the territory

The two files get conflated constantly, and only one of them is in the specification. Only one. llms.txt is the lean index: H1, optional summary, optional link sections. llms-full.txt is not in the specification at all, which is the part that trips people up when they cite it as though the standard blessed it, because the standard did no such thing and instead reaches for a different mechanism entirely. It is an ecosystem convention, popularised by documentation tooling, that concatenates a site's entire content into one Markdown bundle for one-shot ingestion. The specification itself takes a different route to full content: per-page Markdown mirrors, one clean .md file alongside each HTML page. Three surfaces, then. So when someone says a site "has llms.txt," ask which of these three they actually mean.

The famous adopter list is mostly a tooling default

The strongest-looking argument for llms.txt is the roster of serious companies that ship one. Anthropic, Cursor, Stripe, Vercel, and Hugging Face all publish llms.txt files. Impressive list. That looks like a considered industry bet, right up until you trace how the files actually got there and find that a single documentation platform's default setting explains most of the names on it. Mintlify, the documentation platform many of them run, rolled out automatic llms.txt generation on 14 November 2024, per mintlify.com. Much of the celebrated adoption is that default firing, not a deliberate visibility decision. A cascade of auto-generated files is weaker evidence than a list of names implies. The roster is a tooling default, not a verdict.

Does any AI engine actually read llms.txt?

No major AI engine has committed to reading third-party llms.txt files, and the ones that publish their own do so as a byproduct rather than a policy. Google has said so plainly. OpenAI, Anthropic, and Perplexity have said nothing that amounts to a commitment. The one demonstrable consumer in server logs is a coding agent, not an answer engine.

Google: on the record, twice, that it does not use it

Google has been the clearest. John Mueller wrote on Reddit in April 2025 that, as far as he knew, none of the AI services had said they were using llms.txt, and added: "To me, it's comparable to the keywords meta tag," a reference to a signal Google abandoned as spam-prone. Gary Illyes reinforced it at Search Central Live in July 2025, saying Google does not support llms.txt and is not planning to. Googlebot is absent from llms.txt request logs entirely.

The December 2025 Google confusion, and its retraction

There was one wobble worth naming so you can discount it. In December 2025 some Google-owned properties were briefly found hosting llms.txt files, and the SEO community read that as a quiet endorsement. It was not. Mueller clarified publicly that the files were not a signal of Google using llms.txt for ranking or citation, and the files were removed. Google's position through August 2026 is unchanged: it does not read your llms.txt.

OpenAI, Anthropic, and Perplexity: silence, byproducts, and absence

The others resolve quickly. OpenAI has made no statement that GPTBot or OAI-SearchBot reads third-party llms.txt, and GPTBot's 4.51 percent share of llms.txt requests reads as incidental crawling. Anthropic publishes its own llms.txt for its docs at platform.claude.com, but that came through the Mintlify default, not a crawler policy, and ClaudeBot is 0.80 percent of llms.txt requests. Perplexity has said nothing, and PerplexityBot is absent from the log breakdowns. So far, so bleak. Then one name stands out. The one entity that genuinely fetches these files is Claude Code, the coding agent, which out-fetched every retrieval and training crawler in Ahrefs' logs. The real audience for llms.txt is developer tools, not the answer engines you are trying to get cited in.

The numbers: 97 percent of llms.txt files never get fetched

Two studies, two methods, same landing spot: llms.txt is barely fetched and does not measurably move citations. The Ahrefs log study measures whether the file gets read at all. The SE Ranking modelling study measures whether it does anything for the small fraction of sites where it is.

The Ahrefs study: 97 percent of files never get fetched

Louise Linehan and Xibeijia Guan of Ahrefs analysed 137,210 domains against May 2026 server logs, published in June 2026, per ahrefs.com. The headline finding is blunt: 97 percent of llms.txt files received zero requests of any kind. Not zero AI requests. Zero requests, full stop. Of the 3 percent that saw any traffic at all, the largest single fetcher was not an answer engine and not a training crawler but the class of SEO audit tools running site scans, which took 21.7 percent of those requests, ahead of the AI retrieval bots on 19.5 percent. GPTBot sat at 4.51 percent, ClaudeBot at 0.80 percent, and DeepseekBot at 0.02 percent. A file that almost nobody requests cannot be doing much work.

The SE Ranking study: no measurable citation effect

The second study asked the harder question. Not who fetches the file, but whether it changes the outcome for the sites that have one, which SE Ranking tested by analysing roughly 300,000 domains, published on 20 November 2025 via Search Engine Journal, and controlling throughout for authority, schema, and recency so the file's contribution could be isolated from everything that usually travels with it. Their machine-learning model's accuracy actually improved when the llms.txt feature was removed. Noise, in other words. Their conclusion: llms.txt "doesn't seem to directly impact AI citation frequency. At least not yet." If a controlled model gets better at predicting citations by ignoring your llms.txt, the file is not your lever.

What the same data says does move citations

The other side of the SE Ranking study is where the effort points. What correlated, and by how much: In the same 300,000-domain dataset, FAQPage schema correlated with roughly 34 percent more Perplexity citations and 28 percent more in ChatGPT, and ClaimReview markup on statistic-dense content correlated with about 41 percent more citations in Google AI Mode, which together tell you that answer engines reward pages that pre-structure their claims into extractable question-answer and fact-review units the machine can lift without guessing. Organization sameAs added around 22 percent. Speakable added around 18 percent in AI Mode. These are structured-data and extractable-content levers, which is to say they are the machine-readable substance of the page rather than a courtesy note filed at the root, and they are where every scrap of measured signal in that dataset actually lived.

Our experience matches the studies. Our llms.txt has never been our citation bet. We build it, we ship it, and the effort we actually invest goes into schema and clean extractable copy, the levers the 300,000-domain data says move AI mentions. The map file is hygiene. If you want the mechanism behind those levers, the Generative Engine Optimisation guide walks the schema architecture, and the brand citation measurement stack covers how to attribute a citation to a cause rather than a guess.

Why does Leapbuzz still ship llms.txt?

We ship it because the marginal cost is zero and the downside is nothing, not because we think it earns citations. Our build regenerates the whole stack on every deploy, so shipping a correct file is a settled decision rather than a recurring choice. The trick is to ship the format correctly, because a lazy stub is worse than useless.

The build already does it, so the decision is made

In practice: our scripts/build.py regenerates three surfaces on every single deploy: /llms.txt, /llms-full.txt, and a per-page /index.md Markdown mirror alongside every canonical page. The /llms.txt file follows the AnswerDotAI format properly: an H1 with the brand name, a blockquote that summarises what Leapbuzz is, then H2 link sections for home, services, platforms, industries, and blog. None of that is manual labour, and that distinction is the whole argument, because the moment a correct file requires a human to hand-maintain it the economics flip and the effort stops being worth the negligible return the studies describe. Once the generator is written, the cost of keeping a correct file live is a rounding error. That is the entire economic case. Zero effort, zero downside, small upside for the coding agents that do read it.

Ship the correct format, not a linkless stub

One failure mode, and it is common: shipping a file that technically parses but says nothing. The specification requires only the H1, so a file that is nothing but an H1 with no summary and no links is technically valid and practically a stub, the sort of box-tick that satisfies a validator while telling a reader precisely nothing about what the site is or where its useful pages sit. Lighthouse 13.4 flags present-but-linkless llms.txt files, so the minimal-compliant version is also the version a tool will mark against you. Ship the full correct shape: H1, blockquote summary, H2 link sections. It takes no longer to generate than the stub, and it is the version that is actually legible to whatever does read it.

The real 2026 standards fight is in robots.txt

The real 2026 fight over AI content access is inside robots.txt, the file every crawler already fetches and already obeys. Two separate efforts are converging there, and both will matter to your content long after the llms.txt debate has faded. Cloudflare Content Signals go live for every Cloudflare user on 15 September 2026, one month after this post. The IETF is drafting a competing vocabulary. Same base file. A marketing lead should be preparing for the first of those now.

Cloudflare Content Signals: live 15 September 2026

Cloudflare announced Content Signals in March 2026 and set them effective 15 September 2026 for all Cloudflare users, per TechCrunch. The mechanism is a Content-Signal field inside robots.txt with three categories: search, ai-input, and ai-train. A publisher can state that content may be used for search but not for AI training, for example. This matters more than llms.txt for one structural reason: it rides the robots.txt file that crawlers already fetch and already honour, rather than a separate file that 97 percent of the time nobody requests.

IETF AIPREF: drafts only, so far

The IETF is on the same track. The AIPREF working group, launched in January 2025, is drafting a standard vocabulary for AI-usage preferences; it produced working drafts through 2025, per Search Engine Land. As of August 2026, nothing is finalised. Drafts only. What matters for planning is the direction, and the direction is unambiguous once you line the two efforts up side by side: AIPREF, exactly like Cloudflare Content Signals, is built on robots.txt rather than a new file at the root. Both bet against the separate-file model that llms.txt represents. The industry is quietly consolidating AI access control back into the one file crawlers have honoured for thirty years.

What a marketing lead should actually do before 15 September

Turn the abstract into a checklist with a deadline attached. Three moves, in order.

1. Decide your Content Signals policy before 15 September 2026. Work out, per content type, whether you want to permit search, ai-input, and ai-train. A publisher protecting premium content will differ from a brand chasing citation reach. This is a business decision, not a technical one, and it needs an owner.
2. Keep shipping a correct llms.txt if your build already does. It costs nothing and serves the coding agents that read it. Do not add manual effort, and do not expect citations from it.
3. Move your real budget to schema and extractable content. FAQPage, ClaimReview, sameAs, and Speakable are the measured levers from the SE Ranking data. That is where the citation work belongs. The visibility optimisation service is how we run this end to end across the five markets.

llms.txt is cheap insurance and a courtesy to coding agents, not a citation lever. No engine committed to reading it, 97 percent of files never get fetched, and a 300,000-domain model got better at predicting citations by ignoring it. Ship it if your build makes it free, then put your effort where it is measured. At Leapbuzz we run exactly that split across Singapore, Malaysia, Australia, the US, and Canada: correct files at build time, real work on schema and extractable content, and Content Signals policy decided ahead of the September deadline. That is the version of this advice I would stake a client engagement on.