AIFindUsMeasured, not guessed

question

Does llms.txt actually do anything?

No assistant has confirmed it reads llms.txt, and the measured request data is thin. Here is what the evidence supports and what it does not.

author Roydell Clarkepublished 3 min read3 verified sources

On the current public evidence, no — not as a way to get cited. No major assistant has confirmed that it reads llms.txt, and the measured request data is thin. It costs almost nothing to publish, so publish it if you like, but do not count it as work that improves your visibility.

That conclusion is uncomfortable to state in a market that has largely assumed the opposite, so here is the reasoning and its limits.

What does the measured data actually show?

The most useful public measurement to date is a study of llms.txt files across a large crawl, which also looked at the requests those files received.

The headline number is smaller than the discussion around it. Ahrefs reports: "Of the requests that do reach llms.txt files, named AI bots account for 19.5%." So among traffic that arrives at these files at all, roughly a fifth is identifiable AI crawlers, and the rest is something else — generic tooling, scanners, ordinary bots.

Two things are worth separating carefully here. That figure describes traffic to files that exist. It is not a statement about how many sites publish one, and it is not evidence that an assistant used a file to answer a question. A fetch is a fetch. It tells you a request happened, not what came of it.

Has any assistant confirmed it uses llms.txt?

Not publicly, as far as the documentation goes — and that absence is the important part. Vendors do document the mechanisms they actually use.

OpenAI describes its side of the arrangement in terms of crawlers and robots.txt: "OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI." The published control surface is the robots file and the named user-agents. llms.txt does not appear in that role.

Compare that with robots.txt, which has a specification, decades of implementation and first-party documentation. Google Search Central states its function directly: "A robots.txt file tells search engine crawlers which URLs the crawler can access on your site." That is what a supported mechanism looks like when you write it down.

So why do so many sites publish one?

Because it is cheap, it is easy to generate, and the argument for it is plausible. A single markdown file summarising your best content, in a predictable location, sounds obviously useful. Plausible is not the same as demonstrated.

There is also a self-reinforcing loop. Tools generate the file, articles recommend the file because tools generate it, and audits reward the file because articles recommend it. None of those steps is evidence.

Should I publish one anyway?

It is a defensible thing to do, with two conditions.

  • Cost it honestly. Publishing costs minutes. Keeping it accurate as your site changes

costs more, and a stale file describing pages you deleted is worse than no file.

  • Do not let it displace real work. If a morning goes into llms.txt while your content

still needs JavaScript to be visible, that morning was spent on the wrong thing.

If you do publish it, remember that a rule permitting /llms.txt does not permit /llms-full.txt — robots.txt matching is by prefix, and those two paths do not share one.

What this does not tell you

It does not tell you llms.txt is useless, and that distinction matters. Absence of published confirmation is not proof of absence; an assistant could read these files without saying so, and the position stated here would change the moment a vendor documents support or a study links files to citations. Until then the honest summary is: unproven, cheap, and not where your effort belongs.

sources

Each passage was fetched from the publisher’s own page and checked to appear there word for word before this post was written. Open any link and search for the sentence — that is the whole point of printing it.

  1. Ahrefs

    01
    Of the requests that do reach llms.txt files, named AI bots account for 19.5%.

    https://ahrefs.com/blog/llmstxt-study/retrieved 2026-08-04

  2. Google Search Central

    02
    A robots.txt file tells search engine crawlers which URLs the crawler can access on your site.

    https://developers.google.com/search/docs/crawling-indexing/robots/introretrieved 2026-08-04

  3. OpenAI

    03
    OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI.

    https://platform.openai.com/docs/botsretrieved 2026-08-04

how this was written

The sources above were fetched and verified by AIFindUs, then drafted with claude-opus-5 from those passages only, and reviewed and published by Roydell Clarke. A draft that cannot cite a primary source for a claim is not published.

read next