What robots.txt mistakes quietly block AI search?

The three that do the most damage: assuming groups inherit, assuming paths match exactly, and assuming line order decides. All three produce a file that reads correctly to a human and behaves differently in practice.

Each one is worth understanding directly, because each has caused real sites to disappear from answers while looking perfectly configured.

Do robots.txt groups inherit from the wildcard group?

No, and this is the mistake with the widest blast radius. A crawler obeys the most specific group that matches its name, and only that group. Rules under User-agent: * do not apply to a crawler that has its own group.

So this file does not do what it appears to:

User-agent: *
Disallow: /admin/
Disallow: /cart/

User-agent: GPTBot
Allow: /

The intent is to keep everyone out of admin while welcoming GPTBot. The effect is that GPTBot has no Disallow at all, because it read its own group and stopped. If you want hygiene rules to apply to a named crawler, repeat them in that crawler's group. Every time.

Are paths matched exactly?

No. Matching is by prefix, which quietly widens or narrows a rule beyond what was intended. Allow: /llms.txt also permits /llms.txt.bak, and — the case that catches people — does not permit /llms-full.txt, because that path does not start with the same string.

When two rules both match, length decides. IETF RFC 9309 states it with an example: "The following example shows that in the case of two rules, the longest one is used for matching." Not the first line, not the last, not the most specific-looking. The longest matching path.

The practical consequence: adding a broad Disallow at the bottom of the file does not override the specific Allow above it. Many people expect the opposite.

What does robots.txt actually control?

Access by cooperating crawlers, and nothing else. Google Search Central puts the scope plainly: "A robots.txt file tells search engine crawlers which URLs the crawler can access on your site."

Two things follow from that sentence. It is not a security boundary — a URL you disallow is still public. And it is not enforcement — a crawler that ignores the file simply ignores it, and at least one prominent crawler is documented as doing so.

Which crawler names should be in the file?

The ones the vendors actually publish. Inventing plausible tokens is common and useless: a rule for a name nobody sends matches nothing.

OpenAI names its own crawlers and their purposes: "OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI." Two names, two jobs — answer-time retrieval and training collection. Decide on each separately, because blocking the retrieval bot removes you from answers, while blocking the training bot does not.

A short checklist

• Repeat hygiene Disallow lines in every named group.
• Remember that longest-prefix wins, and that your line order is irrelevant.
• Declare your sitemap with a Sitemap: line.
• Never block the scripts, styles or JSON your pages need to render.
• Re-read the file after any migration. Copied robots files outlive their purpose.

What this does not tell you

It does not tell you that a correct robots.txt makes you reachable. Your CDN or firewall can refuse a crawler that your robots file welcomes, and the file has no say in that. The only way to know is to request your own pages with each crawler's user-agent and read the status code you get back.