Which AI crawlers can actually reach my site, and how do I check?

Check by requesting your own pages with each crawler's user-agent string and reading the status code. A 200 means it got in. A 403 or 429 means it did not, whatever your robots.txt intended. Your robots file states a preference; the status code is the outcome.

Those two disagree more often than site owners expect, and the disagreement is usually invisible from inside the company.

How do I test it myself?

One line per crawler, and the answer is unambiguous:

curl -sI -A "OAI-SearchBot/1.0" https://example.com/ | head -1
curl -sI -A "PerplexityBot/1.0" https://example.com/ | head -1
curl -sI -A "ChatGPT-User/1.0" https://example.com/ | head -1

Run it against a real content page, not just the homepage. Protection rules are often path-specific, so / can pass while /pricing refuses.

Which tokens should you test? Use the names the vendors publish rather than guesses. OpenAI documents its own: "OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI." Those are two different jobs — one fetches at answer time, one collects for training — and blocking them has two different consequences.

Why would a crawler be blocked when robots.txt allows it?

Because robots.txt and your edge are separate systems, and only one of them can actually refuse a request.

Google Search Central describes the file's role: "A robots.txt file tells search engine crawlers which URLs the crawler can access on your site." It is an instruction to a cooperating crawler. It is not enforcement, and it is not the only thing in the request path.

A CDN, a firewall or a bot-management product sits in front of your origin and makes its own decision. Turning on an aggressive bot rule, or a managed ruleset that treats unfamiliar agents as suspicious, produces a 403 while your robots.txt still says Allow: /. Both are working as configured. The result is that you are unreachable to the crawler and your configuration says otherwise.

This is the single most common blocking cause we see, and it is invisible unless someone tests by status code.

What should I do with the results?

Sort the outcomes into three buckets.

• Answer-time retrieval blocked. This removes you from that assistant's answers. Fix it

first; nothing else you do to the page matters while it stands.

• Training collection blocked. A legitimate choice with a real trade-off, and one you

should make deliberately rather than discover.

• Everything returns 200. Good. Re-test after any change to your CDN or bot rules,

because that is when this silently breaks.

Keep the raw output. A screenshot of curl -sI beside a robots.txt that permits the same crawler is the fastest way to get an infrastructure change approved.

Does the standard settle any of this?

For matching rules, yes, and the details bite. IETF RFC 9309 gives the resolution rule with an example: "The following example shows that in the case of two rules, the longest one is used for matching." Prefix length decides, not the order of your lines — which is why a rule that looks like it permits something often does not.

What this does not tell you

A 200 proves the crawler could fetch the page. It does not prove anyone quoted it, and no public data would let you claim otherwise. Access is a prerequisite you can verify; citation is an outcome you cannot. Treat the first as a checklist and the second as a hope.