97% of llms.txt Files Are Never Read. Here's the Data.

Ahrefs ran the obvious experiment, which is to check server logs and see whether anyone actually requests the file.

That's not a subtle signal that the tactic underperforms. It's the file sitting on the server, untouched, while people bill for having created it.

The reason is straightforward, and Ahrefs states it plainly: no major LLM provider currently supports llms.txt. Not OpenAI. Not Anthropic. Not Google. None of them has formally adopted it as part of their crawler protocol. A standard that nobody has agreed to implement is a proposal, not a standard.

You don't have to infer Google's position from crawl logs. It's in the documentation, in the AI optimisation guide revised on 10 July 2026.

Google's wording: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search." The guide goes further and says that creating such files will "neither harm nor help your site's visibility or rankings."

Read that last clause carefully, because it's the fairest summary of the whole tactic. Publishing an llms.txt file isn't dangerous. It costs you a few minutes and a few kilobytes. It just doesn't do anything, and the confidence with which it gets recommended is wildly out of proportion to the evidence behind it.

A standard nobody has agreed to implement is a proposal.

You'll see adoption charts showing llms.txt spreading quickly, and they're accurate. They're also measuring something other than what they appear to measure.

A large share of that growth is platform-injected. When a hosting platform ships llms.txt across every store on its network, adoption jumps by tens of thousands of domains overnight without a single site owner deciding anything. The chart goes up. The number of people who chose the tactic doesn't move.

This is worth internalising as a general pattern, because it will happen again with the next proposed standard. Adoption counts measure what platforms ship by default. They tell you nothing about whether the thing works, and nothing about whether anyone on the other end is listening.

There's a narrow case worth acknowledging, because blanket dismissals age badly.

If you publish developer documentation that people feed directly into coding assistants, a curated plain-text index of your docs has genuine utility. The consumer there isn't a search crawler discovering your file autonomously. It's a human pointing a tool at a URL you gave them. That's a real workflow, and some documentation sites support it well.

That is a different thing from an AI visibility tactic. It doesn't get you cited in AI Overviews, it doesn't affect what ChatGPT retrieves, and it isn't what the people selling llms.txt audits are claiming.

Everything below has evidence behind it and is covered in more depth elsewhere in this series.

• Check your crawler access properly. Whether AI systems can reach you at all is decided at the edge, in your CDN settings and your WAF, not in a file you add. That's the Cloudflare category split and the Common Crawl audit.
• Make your existing HTML retrievable. Passage-level structure, a clear claim at the top of each section, no key content locked behind JavaScript. The retrieval systems read your pages, not a manifest describing them.
• Get your robots.txt right. It's the file that AI crawlers genuinely respect, and on Cloudflare it's now generated from your dashboard policy rather than written by hand.
• Measure what you can. Search Console's Generative AI report and Bing's grounding queries are real data, and both are free.

None of that is as satisfying as adding a file and ticking a box. It's the work that actually determines whether you get retrieved.

The broader point isn't really about llms.txt. It's that this field rewards confident recommendations far more than it rewards checking, and the checking is usually one search away. Ahrefs looked at server logs. Google wrote it in the documentation. Both were available to anyone who went looking before putting it in a deck.