On this page

How AI Crawlers See Your Site: What They Fetch and Read

15 September 2026

A shop owner asks an AI assistant when their own shop is open. The assistant answers with hours taken from a directory listing that has been wrong for two years, and adds that the shop's own website does not state its opening hours. The website does state them. They sit on the home page, in a box that reads "Loading opening hours..." until a script replaces the placeholder with the real text. A browser waits for that script. The thing that fetched the page for the assistant did not.

This article explains what actually arrives at your server when an AI system reads your site, and what leaves again. It covers the three different jobs hiding behind the word "crawler", what a single fetch really looks like on the wire, how the user-agent names are constructed and why two of the best known ones are not crawlers at all, what robots.txt can and cannot say, the user-initiated fetchers that state plainly that they ignore it, how to prove that a visitor claiming to be GPTBot really is GPTBot, and the newer machinery that is replacing the whole arrangement: cryptographic bot signatures, HTTP 402 pricing, and an IETF working group trying to move the question out of the user-agent name altogether. Every fetch, DNS lookup, IP-range file and robots.txt in this article was run or downloaded on 8 September 2026, and the output is pasted as it came back.

An AI crawler is not a small browser. It is one HTTP request, and it usually stops where your JavaScript begins.

Goal: after reading this you can tell which machine fetched which page, whether it was really who it claimed to be, what it could read when it got there, and which of your controls it was ever going to obey.

1. The Basics

"AI crawler" is one phrase covering three jobs that have almost nothing in common. They arrive with different names, they answer to different rules, and they produce different consequences for you. Until you separate them, every decision about blocking or allowing is made blind.

1.1 Three Jobs, Not One

Job	What it does	Example tokens	Does robots.txt apply?	What it produces
Training	Collects pages that may be used to build or refine a model	`GPTBot`, `ClaudeBot`, `CCBot`, `Meta-ExternalAgent`	Yes, by vendor policy	Weights. No link, no visit, nothing you can measure later
Search indexing	Builds the index the assistant searches when somebody asks a question	`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `Bingbot`, `Meta-WebIndexer`	Yes, by vendor policy	Eligibility to be retrieved and cited
On-demand fetch	Reads one page, right now, because a person just asked something	`ChatGPT-User`, `Claude-User`, `Perplexity-User`, `Meta-ExternalFetcher`	Often not. See section 5.2	One answer for one person, sometimes with a link

The same company runs several of these, under separate names, with separate rules. Anthropic's own documentation is explicit that the three are independent: blocking ClaudeBot does not block Claude-SearchBot or Claude-User. OpenAI splits the same way, and adds a fourth token, OAI-AdsBot, that checks pages submitted as ads.

So "we blocked the AI bots" is never a complete sentence. Blocked which job? On most sites the three deserve three different answers: no to training, yes to indexing (that is how you get cited at all), and a shrug at the on-demand fetcher, because it is going to fetch the page anyway.

1.2 What One Fetch Actually Looks Like

To show the difference between a fetcher and a browser rather than assert it, here is a small page served locally. It contains one server-rendered sentence, one line filled in later by JavaScript, a stylesheet, a script and an image. First, a fetch with the exact user-agent string OpenAI documents for GPTBot:

$ curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; \
    GPTBot/1.4; +https://openai.com/gptbot" http://127.0.0.1:8731/index.html
<!doctype html>
<html lang="en"><head><meta charset="utf-8"><title>Opening hours</title>
<link rel="stylesheet" href="/style.css"></head>
<body>
<h1>Nijmegen workshop</h1>
<p>Server-rendered: the workshop is at Waalkade 1, Nijmegen.</p>
<div id="hours">Loading opening hours...</div>
<img src="/logo.png" alt="logo">
<script src="/app.js"></script>
</body></html>

That is the whole visit. The server log has one line in it:

127.0.0.1 "GET /index.html" ua=Mozilla/5.0 AppleWebKit/537.36 (KHTML, l

Now the same page in headless Chromium, driven by Playwright, with the text read out of the rendered page:

$ node browse.js
Nijmegen workshop

Server-rendered: the workshop is at Waalkade 1, Nijmegen.

Client-rendered: open Tuesday to Saturday, 09:00 to 17:00

And the log for that single page view:

127.0.0.1 "GET /index.html" ua=Mozilla/5.0 (X11; Linux x86_64) AppleWeb
127.0.0.1 "GET /style.css" ua=Mozilla/5.0 (X11; Linux x86_64) AppleWeb
127.0.0.1 "GET /logo.png" ua=Mozilla/5.0 (X11; Linux x86_64) AppleWeb
127.0.0.1 "GET /app.js" ua=Mozilla/5.0 (X11; Linux x86_64) AppleWeb
127.0.0.1 "GET /hours.json" ua=Mozilla/5.0 (X11; Linux x86_64) AppleWeb

One request against five. The opening hours against a placeholder. That difference is the whole subject of this article, and section 4.3 turns it into a test you can run on your own log.

The request headers are just as thin. This is everything the curl fetch sent, printed by the server:

Host: 127.0.0.1:8731
User-Agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Accept: */*

No cookie, so no session and no logged-in state. No Referer, so nothing about where the request came from. No Accept-Language, so a site that picks a language from that header has to guess. No Sec-Fetch-* headers, no client hints, no Accept-Encoding unless the client bothers. Anything your site does with those inputs, it is doing without them here. For the full chain a normal browser request goes through, see what actually happens when you enter a URL.

A search crawler fetches your page so it can send you a visitor later. An AI fetcher usually fetches your page so it can answer the question itself. Both are worth serving, but only if you know which one is at the door.

1.3 The Outcome Is a Citation, Not a Click

The exchange that built the web was simple: a crawler takes your content, an index sends you readers. AI retrieval keeps the first half and weakens the second. The answer is assembled from your words, and the link is a footnote the reader may never press.

That shift is why the crawl itself is now an economic question rather than a technical one. In its press release of 2026 about the change described in section 7.2, Cloudflare says that "automated agents and bots drive more than half of all web requests", and that "over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages". Both halves matter to a site owner: the traffic is real load, and a large part of it produces nothing new for either side.

None of that means you should block. It means you should know which job is being done, what it costs you, and what it returns. Sections 5 and 6 are about answering that from your own data instead of from somebody's blog post. This article stays on the mechanism, the fetch and the identity behind it; for the separate question of how to write pages that an answer engine can quote well, see the article on GEO, AEO and AIO.

1.4 How the URL Was Found in the First Place

Before any of this, the machine has to know your URL exists. Discovery is the step nobody thinks about until a page gets no fetches at all, and the paths are the familiar ones: links from pages the crawler already has, an XML sitemap, an existing search index, a feed, a link from another site. An assistant that searches before it answers does not usually discover you at all; it starts from an index somebody else built, which is why being indexed and being fetched are two separate wins.

One discovery path is specific to this subject: a person pastes your URL into a prompt. Nothing was crawled, nothing was indexed, no link pointed anywhere. That is the whole reason the on-demand fetchers of section 1.1 exist, and it is why there is no discovery work you can do to influence them.

The Sitemap line at the foot of a robots.txt is worth one precise note, because it is not part of the protocol at all. RFC 9309 files it under "Other Records": crawlers "MAY interpret other records that are not part of the robots.txt protocol", and the only firm rule is that such a record "MUST NOT terminate a group". So a Sitemap line sitting in the middle of your file does not split the group it landed in, and a crawler is within its rights to ignore it entirely.

2. Where the Names Come From

The user-agent names look like a random collection of brand words, but they follow a grammar, and the grammar tells you what the thing does before you look anything up.

2.1 The Four Name Patterns

Pattern	Examples	What it means
`<Vendor>Bot`	`GPTBot`, `ClaudeBot`, `PerplexityBot`, `Amazonbot`, `CCBot`	An autonomous crawler. It decides when to visit. Nobody is waiting for the page
`<Vendor>-SearchBot`	`OAI-SearchBot`, `Claude-SearchBot`	A crawler whose output is the product's own search index rather than training data
`<Vendor>-User`	`ChatGPT-User`, `Claude-User`, `Perplexity-User`, `Meta-ExternalFetcher`	A fetch triggered by a person who is waiting for the answer. This is the group that treats `robots.txt` as advisory
`<Vendor>-Extended`	`Google-Extended`, `Applebot-Extended`	Not a crawler at all. A name that exists only so you can write a rule about it

The last row is the one that catches people, and both vendors say so plainly. Apple writes that "Applebot-Extended does not crawl webpages" and that it "is only used to determine how to use the data crawled by the Applebot user agent". Google writes that Google-Extended "doesn't have a separate HTTP request user agent string" and that crawling "is done with existing Google user agent strings".

So these two tokens are switches, not visitors. You will never see either of them in an access log, because neither of them ever sends a request. People wait for a log line that cannot appear, conclude the rule is not working, and remove it. The rule was working; the name simply is not a machine.

Google adds a second consequence worth knowing: disallowing Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". You can opt out of the training use without paying for it in search.

2.2 The Words Themselves

GPTBot takes its name from GPT, generative pre-trained transformer, the model family, so the name describes the purpose of the collection rather than the crawler.
CCBot is Common Crawl, a non-profit that has been publishing an open crawl of the web since long before the current AI wave. Its archive is a training source for many models, which is why one Disallow aimed at CCBot affects more companies than any other single rule in your file.
OAI- is simply OpenAI shortened, and it prefixes the two tokens that are not the historical GPTBot: OAI-SearchBot and OAI-AdsBot.
Meta's names are the most descriptive of the set, and they changed: alongside Meta-ExternalAgent for "training foundation AI models or improving products by indexing content directly", the documentation now lists Meta-WebIndexer, whose stated job is to "improve Meta AI search result quality", and Meta-ExternalAds.

One naming trap survives from the early period. Anthropic's first tokens were anthropic-ai and Claude-Web; both are retired, and rules written against them now do nothing. Old blog posts still recommend them, and old robots.txt files still carry them. If your file names either, it is stale, and section 9 lists the other fossils to look for.

3. A Short History

Every control you have over AI crawling is bolted onto a file format that was agreed by email in 1994, for a different problem. Knowing that history explains most of the awkwardness.

When	What happened	Why it still matters
30 June 1994	A Standard for Robot Exclusion, agreed "on the robots mailing list" between "the majority of robot authors and other people with an interest in robots"	The document says of itself: "It is not an official standard backed by a standards body ... It is not enforced by anybody"
September 2022	RFC 9309, Robots Exclusion Protocol, Standards Track	28 years later the convention finally has precise rules for matching, caching, size limits and error handling. Section 5.3 depends on them
2023	`GPTBot` and `Google-Extended` appear	The first tokens to separate "may you read this" from "may you train on this". The file was never designed to express that difference
September 2023	Bing adds `NOCACHE` and `NOARCHIVE` as AI controls	The first controls that live in a meta tag rather than in `robots.txt`, because the question is about use, not access. See section 5.4
2024	`ClaudeBot`, `Applebot-Extended` and `Meta-ExternalAgent` arrive, and the `-User` fetchers become a documented category of their own	The three-job split of section 1.1 becomes visible in the names
3 September 2024	Jeremy Howard proposes `llms.txt` (version 2 followed on 10 August 2026)	A publishing convention that spread widely among documentation sites without any crawler committing to read it. Measured in section 7.4
1 July 2025	Cloudflare changes its default to block AI crawlers for new domains and opens a pay-per-crawl beta	For the first time the answer a crawler gets may be decided by your CDN rather than by your file
2025 and 2026	The IETF's AIPREF working group takes up the problem; the vocabulary draft reaches revision 06 (28 April 2026) and the attachment draft revision 05 (19 August 2026)	An attempt to move the preference out of the user-agent name and into a vocabulary that updates RFC 9309. Section 7.3
15 September 2026	Cloudflare splits its AI bot switch into Search, Training and Agent, and gives new ad-carrying domains a preset that disallows training and blocks agents on ad pages	The three-job split of section 1.1 becomes a setting. Existing sites keep what they had, so what changed is the vocabulary rather than your defaults. Section 7.2

Read down the "why it matters" column and one pattern stands out. The 1994 file answers exactly one question: may you fetch this path? Every question that has been asked since 2023 is a different kind of question. May you train on it. May you remember it. May you quote it in an answer that replaces the visit. None of those are fetch questions, and all of them are currently expressed by inventing another name to disallow.

The reason AI crawler control feels improvised is that it is. We are answering questions about use with a file that can only talk about access.

4. Simple Use Cases: Seeing What They See

Four checks, each one command long, tell you more about your AI visibility than any dashboard. They answer: what does a fetcher receive, does it contain the words you think it does, who has been fetching, and is your robots.txt even readable.

4.1 Fetch Your Own Page the Way a Fetcher Does

Use curl with the documented user-agent string of the bot you care about. You are not pretending to be OpenAI; you are reproducing the request shape, which is what matters. Nothing here is authenticated, so what you get is what any anonymous client gets:

$ curl -sS -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; \
    GPTBot/1.4; +https://openai.com/gptbot" \
    https://petermartin.nl/en/focus-on/web/dns \
    -o page.html -w "%{http_code} %{size_download} bytes\n"
200 197609 bytes

A 200 and a body of the size you expect is the first half of the answer. The second half is whether the words are in it, which is a question about text and not about bytes: an empty shell of an application can be 180 KB of JavaScript and no sentences at all.

Do not expect the byte count itself to be reproducible. Run the same command twice and it moves a little. On this site the page ends with a module that picks a few related articles at random, and those cards differ in length because the titles, the image filenames and the generated thumbnail URLs differ in length: about a hundred bytes either way, on a page of roughly 190 KB. The website also answers anonymous visitors with Cache-Control: no-store, so every request is a fresh render rather than a cached copy, and anything random on the page is drawn again. The figure above is one capture of a page that also grows as it is edited. What you are reading is the order of magnitude, not the digits.

4.2 The Rendering Test

Strip the tags from what you just downloaded and count the words. On the same page, that gives:

words in raw HTML text: 17456
first 200 chars: What Is DNS? How It Works and What Every Record
Type Does - Peter Martin Skip to main content Focus on Joomla Linux
The web Presentations Joomla Linux Other About Me Work Volunteer
Nederlands English ...

Seventeen thousand words arrived in the first response, before a single script ran. That is what a server-rendered page looks like from the outside, and it is the answer you want.

The failing version looks like the demo in section 1.2: the fetcher receives Loading opening hours... where the browser shows the hours. To test your own site for that, compare the two numbers. Count the words in the raw HTML, then count the words in the rendered page (in the browser console, document.body.innerText.split(/\s+/).length). If the rendered count is much larger, the difference is the part only browsers can see.

What do the crawlers actually do with JavaScript? The honest answer is that only some vendors say. Google documents that Googlebot renders pages, and its AI features are built on that same crawling and indexing infrastructure. OpenAI, Anthropic and Perplexity all publish crawler documentation, and none of it makes any claim about executing JavaScript at all, in any direction. When the vendor is silent, do not assume in your favour: measure your own logs with the test in the next section, and in the meantime make sure the words you want quoted arrive in the first response.

4.3 Find Them in Your Access Log

Every fetch leaves a line. Counting by token tells you who reads you and how much:

$ grep -oiE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|\
PerplexityBot|Perplexity-User|CCBot|Bytespider|Amazonbot|meta-externalagent|Applebot' \
    access.log | sort | uniq -c | sort -rn

Then use the fingerprint from section 1.2 to ask whether a visitor rendered the page. Group a single visitor's requests by second and look at what followed the HTML:

$ awk '$0 ~ /GPTBot/ {print $1}' access.log | sort -u > bot-ips.txt
$ grep -F -f bot-ips.txt access.log | awk '{print $7}' | sort | uniq -c | sort -rn

Then group the same visitor's requests by the code you answered with, because a crawler that is being refused will never tell you so:

$ awk '/GPTBot/ {print $9}' access.log | sort | uniq -c | sort -rn

A wall of 403 means something in front of your application is turning the crawler away, which is section 7.2. A row of 429 means you are rate limiting it. A row of 5xx means it caught your site at a bad moment, and if one of those requests was for robots.txt, the next section explains what that did to the rest of your site.

If that list contains only HTML URLs, the client never fetched a stylesheet, a script or an image, which means it never rendered anything. If you see .css, .js and image requests from the same address in the same seconds, something with a rendering engine came to visit. This test costs nothing, it uses data you already have, and unlike a vendor statement it is about your site.

4.4 Check That Your robots.txt Answers Correctly

The file has to be reachable, at the root of the host, and it has to answer with a success code. RFC 9309 is precise about what happens when it does not, and the two failure modes are opposites:

What the server answers	What the crawler must assume
`2xx`	The rules in the file
`4xx` ("unavailable")	"the crawler MAY access any resources on the server"
`5xx` ("unreachable")	"the crawler MUST assume complete disallow"
More than five redirects	Crawlers "MAY assume that the robots.txt file is unavailable"

A missing file opens everything; a broken server closes everything. An hour of 503 during a deploy is therefore not neutral: a crawler that reads your robots.txt in that hour must treat your whole site as disallowed, and may keep that cached. The RFC says crawlers "SHOULD NOT use the cached version for more than 24 hours, unless the robots.txt file is unreachable", so an outage can outlive itself.

The vendors describe the same delay from their side. OpenAI's crawler documentation states that "it can take ~24 hours from a site's robots.txt update for our systems to adjust". A change to that file is a request for a change tomorrow, not a switch you just flipped.

$ curl -sS -o /dev/null -w '%{http_code} %{content_type} %{size_download}\n' \
    https://petermartin.nl/robots.txt
200 text/plain 979

Two more limits from the same document are worth remembering: the product token may only contain letters, underscores and hyphens, and matching is case-insensitive, so gptbot and GPTBot are the same rule. And crawlers must parse at least the first 500 kibibytes, which is far more than any sane file needs, but not infinite for the generated monsters some plugins produce.

5. Moderate Use Cases: Deciding What They May Do

Now the decisions. The order below matters: the first two subsections decide what is even possible, and the rest is detail.

5.1 One Rule per Token, per Job

A rule applies to the token you name, and to nothing else. This file allows the search crawlers, refuses the training crawlers, and says nothing about the on-demand fetchers:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://example.com/sitemap.xml

Read it against the table in section 1.1 and you can see exactly what it buys: no training use, full eligibility to be found and cited. That is the posture most business sites actually want, and it is not the posture either "block the AI bots" or "do nothing" gives you.

5.2 The Fetchers That Say They Will Not Obey

This is the single fact that changes how you plan. The user-initiated fetchers do not promise to follow robots.txt, and their documentation says so in plain language:

OpenAI on ChatGPT-User: it is "used for certain user actions in ChatGPT and Custom GPTs", and because these are user-initiated actions, "robots.txt rules may not apply".
Perplexity on Perplexity-User: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules."
Meta on Meta-ExternalFetcher: it "may bypass robots.txt rules" because it performs fetches requested by the user. Meta says the same of facebookexternalhit when it is "performing security or integrity checks".

The reasoning is consistent, and it is the same reasoning that has always exempted a browser: a person asked for this page, and a person is allowed to read your page. Whether the person is holding the mouse or holding a prompt does not change the request.

The practical consequence: robots.txt governs bulk collection, not individual reading. If a page must not be read by anyone who does not belong there, the control is authentication, not a text file. Nothing in this article replaces a login.

5.3 The Trap in Group Precedence

RFC 9309 says that a crawler finds the group matching its own product token, merges any duplicate groups for that token, and only if there is none "MUST obey the group with a user-agent line with the '*' value". A specific group does not add to the * group. It replaces it.

That produces a failure that looks like generosity. Here is the real robots.txt of wordpress.org, fetched on 8 September 2026, shortened to the relevant lines:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: *
Disallow: /search
Disallow: /?s=

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

The intent is clear and friendly: AI crawlers are welcome. The effect is that GPTBot, ClaudeBot and PerplexityBot no longer match the * group, so for those three crawlers the /search and /?s= rules simply do not exist. The site's own search pages, which every generic crawler is kept out of, are open to exactly the crawlers that were given a warm welcome.

The fix is to repeat what you meant, in the specific group:

User-agent: GPTBot
Allow: /
Disallow: /search
Disallow: /?s=

Any time you write a per-bot group, copy your generic Disallow lines into it. Every one you leave out, you have just granted.

Group precedence decides which block of rules applies. A second rule decides which line inside that block wins, and it is not the order they are written in. RFC 9309: "The most specific match found MUST be used. The most specific match is the match that has the most octets." Length decides, and on a tie the permissive line wins: "If an 'allow' rule and a 'disallow' rule are equivalent, then the 'allow' rule SHOULD be used."

User-agent: ExampleBot
Disallow: /private/
Allow: /private/public/

/private/report          blocked   (matched by /private/, 9 octets)
/private/public/report   allowed   (matched by /private/public/, 16 octets)

One asymmetry inside the same document catches people out. Matching the product token is case-insensitive, so gptbot and GPTBot are one rule, but path matching "SHOULD be case sensitive". Disallow: /Private/ does not cover /private/, and on a case-insensitive filesystem both URLs may serve the same page.

5.4 The Controls That Are Not robots.txt

Access is one question; use is another, and the second one is answered elsewhere.

The -Extended tokens. Google-Extended and Applebot-Extended live in robots.txt but control use, not fetching. The page is still crawled, still indexed, still ranked, and excluded from model training and grounding.
Bing's meta directives. Microsoft's announcement of September 2023 defines two levels: content marked NOCACHE "may be included in Bing Chat answers" but only as "URL/Snippet/Title", while content marked NOARCHIVE "will not be included in Bing Chat answers, not be linked to in the answers", and neither is used for training beyond those limits. If a page carries both, Bing treats it as NOCACHE.
Google's AI features are not a separate crawler. This is the most common misconception in the whole subject. Google documents that "AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search", and that to appear as a supporting link in AI Overviews or AI Mode "a page must be indexed and eligible to be shown in Google Search with a snippet". Google-Extended does not govern them. What does limit them is the snippet family: nosnippet, data-nosnippet, max-snippet and noindex. There is no setting that keeps you in Google Search and out of AI Overviews without also giving up your snippet.
X-Robots-Tag. The same directives can be sent as an HTTP response header instead of a meta tag, which is the only way to mark a PDF, an image or any other non-HTML file.

These are not interchangeable with a Disallow, and combining them wrongly cancels them out: a page you disallow in robots.txt is never fetched, so the noarchive in its HTML is never read. The same trap that has always applied to noindex applies here, and it is covered in more detail in the Joomla article on what robots.txt can and cannot do.

5.5 What Publishers Actually Do

Rather than guess at industry practice, here is the industry practice. Each of these robots.txt files was fetched on 8 September 2026 and parsed by the rules of RFC 9309. BLOCK means the token's group contains Disallow: /, partial means it has narrower rules, allow means a group exists with no disallow, and - means the token is not named at all and falls back to *:

site               *        GPTBot   OAI-Sear ChatGPT- ClaudeBot Perplex. CCBot    Meta-Ext
nytimes.com        partial  BLOCK    BLOCK    BLOCK    BLOCK     BLOCK    BLOCK    BLOCK
theguardian.com    partial  -        -        -        BLOCK     BLOCK    BLOCK    BLOCK
bbc.co.uk          partial  BLOCK    BLOCK    BLOCK    BLOCK     BLOCK    BLOCK    BLOCK
reuters.com        BLOCK    -        partial  partial  -         -        -        -
nos.nl             partial  BLOCK    -        BLOCK    BLOCK     BLOCK    BLOCK    BLOCK
wordpress.org      partial  allow    -        -        allow     allow    allow    -
wikipedia.org      partial  -        -        -        -         -        -        -
github.com         partial  -        -        -        -         -        -        -

Three different strategies are visible in eight lines.

The blocklist (New York Times, BBC, NOS): name every AI token you know and refuse it. Complete until a new token appears, which is why these files are long and keep growing.
The allowlist (Reuters): refuse everyone with User-agent: * and Disallow: /, then name the crawlers that are welcome in a single group with 86 User-agent lines. Nothing new gets in by default. This is the only approach that survives a bot you have never heard of.
The invitation (WordPress.org): name the AI crawlers to say yes, with the side effect described in section 5.3.

The Guardian's line is the interesting one. It blocks Anthropic's and Perplexity's crawlers, and does not mention OpenAI's at all, which means GPTBot is allowed. A robots.txt is no longer only a technical file. On a large publisher it records a position, and the position can differ per company for reasons that have nothing to do with crawling. Wikipedia and GitHub, at the other end, name no AI token at all. On sites whose content already carries a licence, the robots file is not where that question gets answered.

5.6 The Header That Outlives the Rule

A rule takes effect when the crawler reads it, and how long that takes is decided by a header you probably never set. RFC 9309 addresses both halves in three sentences: "Crawlers MAY cache the fetched robots.txt file's contents. Crawlers MAY use standard cache control as defined in [RFC9111]. Crawlers SHOULD NOT use the cached version for more than 24 hours, unless the robots.txt file is unreachable."

So ask your own server what it tells them:

curl -sSI https://example.com/robots.txt | grep -i 'cache-control\|expires'

I ran that against this site on 15 September 2026, while writing this article, and got thirty days:

cache-control: max-age=2592000
expires: Thu, 15 Oct 2026 21:58:58 GMT

It was not a decision anyone had made about robots.txt. It was one blanket expires 30d rule in nginx, applied to every static file on the server, and robots.txt is a static file. The favicon wants that rule. The file that governs crawler access does not.

The SHOULD in the RFC caps the damage at 24 hours for any crawler that implements it. It does not cap anything that implements only RFC 9111: a proxy, a CDN edge, an HTTP client library with a cache on it. Those get exactly what the header asked for and hold your old rules for a month.

The asymmetry is the reason to care. Removing a rule takes effect at once, because nothing has to notice. Adding one - the rule you wrote this morning because a new crawler is hammering the site - waits for the cache. An exact-match block overrides the blanket rule without touching it, and this is the fix I applied here:

location = /robots.txt {
    expires 1d;
}

On Apache the same thing, with mod_expires loaded:

<Files "robots.txt">
    ExpiresActive On
    ExpiresDefault "access plus 1 day"
</Files>

Run the curl again and confirm max-age=86400, which is what this site returns now. It is the cheapest fix in this article: one block, one check, and every rule in this section starts working tomorrow instead of next month.

6. Advanced Use Cases: Proving Who Called

Everything in section 5 assumes the visitor is who it says it is. It is not, necessarily. A user-agent string is a line of text the client chooses, and every log line in section 1.2 that says GPTBot was produced by curl on a laptop in Nijmegen. Blocking, rate limiting and reporting all rest on identity, so identity has to be established rather than read.

6.1 Reverse DNS, and the Forward Check People Skip

The classic method, and the one Google, Apple and Microsoft all document, is a two-step lookup. Take the IP address from the log, ask for its PTR record, and check the hostname ends in the vendor's domain. Then look that hostname up again and check it resolves back to the address you started with. The second step is the one that matters: a PTR record is published by whoever controls the address block, so without the forward confirmation anyone can point a PTR at googlebot.com.

$ dig +short -x 66.249.66.1
crawl-66-249-66-1.googlebot.com.
$ dig +short crawl-66-249-66-1.googlebot.com
66.249.66.1

Same address out as in, so this really is Google. The same works for Apple and for Microsoft:

$ dig +short -x 17.241.208.161
17-241-208-161.applebot.apple.com.
$ dig +short -x 40.77.167.1
msnbot-40-77-167-1.search.msn.com.

For the mechanics of PTR records and why the reverse tree is a separate namespace, see how DNS works and what every record type does.

6.2 Why Reverse DNS Fails for the AI Crawlers

Now try the same thing on addresses that Anthropic publishes as its own:

$ dig +short -x 216.73.216.1        # nothing at all
$ dig +short -x 34.162.230.222
222.230.162.34.bc.googleusercontent.com.
$ whois 216.73.216.1 | grep -i orgname
OrgName:        Amazon.com, Inc.

No anthropic.com hostname exists, because the crawler does not run on hardware Anthropic owns. It runs in Amazon and Google cloud regions, and the reverse tree names the cloud, not the tenant. The same is true of Perplexity, whose published addresses resolve to compute-1.amazonaws.com. For these crawlers the reverse-DNS method does not merely fail to confirm; it produces an answer that looks like a forgery when it is genuine.

That is why the vendors publish address lists instead, and why Anthropic warns in the same breath that blocking by address "may not work correctly or persistently guarantee an opt-out": tomorrow the crawler may run in a different cloud region.

6.3 The Published Address Lists

Almost every operator now publishes its ranges as JSON, in the same shape Google invented: a creationTime and a list of ipv4Prefix and ipv6Prefix entries. This table was built by downloading each file on 8 September 2026 and counting what came back.

Operator, and what the file covers	URL	Prefixes	File dated
OpenAI, `GPTBot`	`openai.com/gptbot.json`	21	2025-10-30
OpenAI, `OAI-SearchBot`	`openai.com/searchbot.json`	35	2026-01-02
OpenAI, `ChatGPT-User`	`openai.com/chatgpt-user.json`	207	2026-09-04
OpenAI, `OAI-AdsBot`	`openai.com/adsbot.json`	2	2026-05-12
Anthropic, all three Claude bots	`claude.com/crawling/bots.json`	26	2026-08-18
Perplexity, `PerplexityBot`	`perplexity.com/perplexitybot.json`	8	2025-02-07
Perplexity, `Perplexity-User`	`perplexity.com/perplexity-user.json`	4	2025-10-17
Apple, `Applebot`	`search.developer.apple.com/applebot.json`	33	2026-07-31
Common Crawl, `CCBot`	`index.commoncrawl.org/ccbot.json`	5	2026-08-11
Google, common crawlers including Googlebot	`developers.google.com/static/crawling/ipranges/common-crawlers.json`	317	2026-09-07
Google, special-case crawlers	`developers.google.com/static/crawling/ipranges/special-crawlers.json`	272	2026-09-07
Google, user-triggered fetchers	`developers.google.com/static/crawling/ipranges/user-triggered-fetchers.json`	1058	2026-09-07
Google, user-triggered fetchers on Google IPs	`developers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.json`	496	2026-09-07
Microsoft, `Bingbot`	`bing.com/toolbox/bingbot.json`	28	2024-01-03

Read the last column before the third. Google regenerates its files daily, and the copies downloaded here were one day old. OpenAI's ChatGPT-User file was four days old, which fits a fetcher whose 207 prefixes move around. And then there is Bing's file, dated 3 January 2024: two years and eight months before it was downloaded. Perplexity's crawler list is nineteen months old.

A stale list is worse than no list, because it fails silently and in the wrong direction. Every address the operator has added since the file was written now looks like an impostor to your verification code, so a real crawler gets rate limited or blocked and nothing in your monitoring says why. Whatever you build on these files, log the creationTime you are matching against, and alert when it stops moving.

6.4 A Verification Script You Can Actually Run

Put those two ideas together and bot verification is about thirty lines. This script takes an address from your log and tells you which published list, if any, it belongs to:

#!/usr/bin/env python3
"""Check whether an IP address belongs to a published AI crawler range."""
import ipaddress, json, sys, urllib.request

FEEDS = {
    "GPTBot":        "https://openai.com/gptbot.json",
    "OAI-SearchBot": "https://openai.com/searchbot.json",
    "ChatGPT-User":  "https://openai.com/chatgpt-user.json",
    "ClaudeBot*":    "https://claude.com/crawling/bots.json",
    "PerplexityBot": "https://www.perplexity.com/perplexitybot.json",
    "Applebot":      "https://search.developer.apple.com/applebot.json",
    "CCBot":         "https://index.commoncrawl.org/ccbot.json",
    "Googlebot":     "https://developers.google.com/static/crawling/ipranges/common-crawlers.json",
    "Bingbot":       "https://www.bing.com/toolbox/bingbot.json",
}

def prefixes(url):
    req = urllib.request.Request(url, headers={"User-Agent": "bot-verifier/1.0"})
    with urllib.request.urlopen(req, timeout=20) as r:
        doc = json.load(r)
    nets = [ipaddress.ip_network(p[k]) for p in doc["prefixes"]
            for k in ("ipv4Prefix", "ipv6Prefix") if k in p]
    return doc.get("creationTime", "?"), nets

ip = ipaddress.ip_address(sys.argv[1])
for name, url in FEEDS.items():
    created, nets = prefixes(url)
    hit = any(ip in n for n in nets)
    print("%-14s %-5s %4d prefixes  file dated %s" %
          (name, "MATCH" if hit else "-", len(nets), created[:10]))

Two runs, one address that belongs to Anthropic and one that belongs to Google:

$ python3 verify_bot.py 216.73.216.5
GPTBot         -       21 prefixes  file dated 2025-10-30
OAI-SearchBot  -       35 prefixes  file dated 2026-01-02
ChatGPT-User   -      207 prefixes  file dated 2026-09-04
ClaudeBot*     MATCH   26 prefixes  file dated 2026-08-18
PerplexityBot  -        8 prefixes  file dated 2025-02-07
Applebot       -       33 prefixes  file dated 2026-07-31
CCBot          -        5 prefixes  file dated 2026-08-11
Googlebot      -      317 prefixes  file dated 2026-09-07
Bingbot        -       28 prefixes  file dated 2024-01-03

$ python3 verify_bot.py 66.249.66.1
...
Googlebot      MATCH  317 prefixes  file dated 2026-09-07

One detail from writing it is worth passing on. The first version used Python's default user-agent and Anthropic's file answered 403 Forbidden. Setting any user-agent string fixed it. The lists exist to be read by scripts, and at least one of them is behind protection that dislikes scripts, which is a fair summary of the state of this whole field.

6.5 Where This Is Going: Signed Requests

Reverse DNS is a lookup about an address. An address list is a file about a network. Neither is about the request. The direction of travel is to sign the request itself, using RFC 9421 HTTP Message Signatures, in an IETF draft usually called Web Bot Auth. The crawler holds a private key, publishes the public key at a well-known location, and adds three headers to every request. The shape, with the values shortened:

Signature-Agent: "https://crawler.example"
Signature-Input: sig1=("@authority" "signature-agent");created=1757337600;keyid="..."
Signature: sig1=:ZXhhbXBsZS1zaWduYXR1cmU...:

The server verifies the signature against the published key. Nothing has to be looked up about the network the request came from, nothing goes stale in a JSON file, and a forged user-agent string gets you nowhere because the signature covers parts of the request itself. Cloudflare verifies these signatures in production as part of its verified-bots programme, and several AI vendors sign. The drafts are not finished standards yet, so treat this as the thing to watch rather than the thing to deploy: what you can do today is make sure your edge does not strip the Signature-Agent header before your server sees it.

7. Things Most Site Owners Do Not Know

Four developments that change the picture, and one boundary that no setting can move.

7.1 HTTP 402 Finally Has a Job

402 Payment Required sat in the HTTP specification for thirty years marked as reserved. Cloudflare's pay-per-crawl design uses it for exactly what it was named after. A crawler asks for a page, and instead of the page it gets a price:

HTTP/2 402
crawler-price: USD 0.05

A crawler that is willing to pay repeats the request with crawler-exact-price, or declares a ceiling up front with crawler-max-price on the first request. When the transaction succeeds the response is a normal 200 carrying crawler-charged. The identity that gets billed comes from the signature machinery in section 6.5, which is what makes the whole thing possible: you cannot charge an account you cannot identify. The feature is in private beta, so this is a mechanism to understand rather than a service to sign up for. For the rest of the status code, see what each HTTP status code actually tells a client.

7.2 Your CDN May Already Be Answering for You

On 1 July 2025 Cloudflare began blocking AI crawlers by default for new domains. On 15 September 2026 it split the control itself: the single "Block AI Bots" switch became three, named Search, Training and Agent. That is the table in section 1.1 arriving in a dashboard, with one renaming to watch, because Cloudflare's "Agent" is what this article calls on-demand fetch.

New domains now get a preset chosen by business model. A site carrying ads starts at Search allowed, Training set to "Disallow AI Training", and Agent blocked on the pages that display ads. A site without ads starts with all three allowed. Existing sites were not moved onto that preset; they were migrated from whatever they already had, and a site that never touched the setting stays on Allow. One thing does reach existing free-plan sites, but it is a signal rather than a block: a free-plan domain with no robots.txt of its own is served Cloudflare's Content Signals Policy when a crawler asks for the file, and that policy, in Cloudflare's words, "does not express any specific preferences about your content". It defines a vocabulary and leaves the answer blank. The file a crawler reads at your root may still be one you never wrote.

The real change for existing sites is not a new default but a re-reading of an old choice. A previous Training selection of Block becomes "Disallow AI Training", which is not a block at all. It publishes a no-training preference in your robots.txt and leaves search crawlers such as Applebot, Bingbot and Googlebot allowed. Compare that with the file in section 5.1: the posture recommended there is now the recommended setting of the largest bot-management vendor on the web. Cloudflare's own numbers explain the switch. 17% of its sites block training in some form, while fewer than 1% block search crawlers. Almost nobody who blocked meant to disappear.

Agent is the exception, and the exception is instructive. It has Allow and Block but no Disallow, and Cloudflare's stated reason is that no well-established directive for agent preferences exists to publish. Section 7.3 is the same sentence written from the other end: where a vocabulary exists you can state a preference, and where it does not you are back to deciding who may knock.

Whatever you think of this as policy, note what it means operationally. The answer a crawler receives may be decided one layer above your site, by a default you never chose, on a date somebody else picked, and now also by a re-interpretation of a setting you did choose. Three consequences follow. First, test from outside rather than reading your own file: fetch your page with a crawler user-agent from a network that is not yours and see what comes back. Second, your CDN's bot settings and your robots.txt are two sources of truth for the same question, and "Disallow AI Training" changes the relationship between them, because the CDN now writes into the file rather than merely overriding it. Third, a migration that changes behaviour immediately and the dashboard a week later leaves a window in which the switch you are looking at is not the rule that is running.

The same layer refuses crawlers you wanted, and it does it quietly. Bot protection, a JavaScript challenge and a CAPTCHA do not know that this particular automated client was invited; they see an HTTP client that is not a browser. Two examples turned up while writing this article, both on 8 September 2026 and both from companies that publish crawler documentation. Anthropic's own IP-range feed answered 403 to a script sending Python's default user-agent, as section 6.4 describes. OpenAI's publisher FAQ answered 403 with "Enable JavaScript and cookies to continue" to a command-line fetch that was sending a browser user-agent string. Two vendor pages about crawler access, neither of them readable by a crawler. If it happens to them, assume it can happen to you, and check your own pages the way section 4.1 does rather than trusting that your robots.txt says yes.

7.3 The Preference Is Moving Out of the Name

Everything in section 5 works by naming companies. That does not scale, it goes stale every time a vendor renames a bot, and it forces you to re-answer the same question for each new entrant. The IETF's AIPREF working group is building the alternative: a small vocabulary of usage categories that any operator can read, attached to content in two ways.

The vocabulary draft defines train-ai, "the act of using an asset in the production or refinement of an AI model that can generate content in one or more modalities", and search, use in an application whose primary purpose "is to select assets and direct users to the location of those assets". Each takes y or n, and saying nothing is a third, distinct state. The attachment draft defines where those preferences go: a Content-Usage HTTP response header field, and a Content-Usage rule inside robots.txt. Its abstract is explicit that this "updates RFC 9309 to allow for the inclusion of usage preferences".

Content-Usage: train-ai=n

User-agent: *
Content-Usage: train-ai=n
Allow: /

One statement, addressed to nobody in particular, that says what may be done rather than who may knock. Neither draft is a published standard yet, and nothing obliges a crawler to look at them today, so this is not a replacement for the rules in section 5. It is the shape the answer is likely to take, and the reason to keep an eye on the working group instead of on the next vendor blog post.

7.4 llms.txt: Measured Rather Than Argued

llms.txt is a Markdown file at the root of a site, proposed on 3 September 2024, that lists your most important pages so a model does not have to work them out from your navigation. The idea is sensible. The question is whether anything reads it.

On 8 September 2026 I requested /llms.txt from 21 well-known sites, following redirects and counting only responses that were both 200 and text/plain:

present (5):  github.com  shopify.com  cloudflare.com  vercel.com  wordpress.org
absent (16):  nytimes.com  reuters.com  bbc.co.uk  wikipedia.org  theguardian.com
              nos.nl  tweakers.net  stackoverflow.com  anthropic.com  mozilla.org
              joomla.org  drupal.org  php.net  python.org  ietf.org  w3.org

Every site that has one is a developer platform publishing documentation. No news publisher in the sample has one, and neither does any standards body. That is the shape of a documentation convention, not of a crawler protocol, and the vendor documentation agrees: the crawler pages of OpenAI, Anthropic, Perplexity, Google, Apple, Meta and Common Crawl explain user agents, robots.txt tokens and IP ranges, and none of them says the crawler reads an llms.txt.

The neatest illustration is Perplexity's own documentation site, which serves a banner reading "For AI agents: see the complete llms.txt documentation index" on the very page that documents its crawlers. That banner is a feature of the documentation platform the site is built on. Publishing one and consuming one are different activities, and the gap between them is the entire story of this file.

That scan measures who publishes one. A server log measures whether anything reads the llms.txt file. My petermartin.nl site has served an llms.txt since 5 July 2026. In the 73 days from then to 15 September 2026 it answered 701,693 requests, of which 43,620 came from user agents naming one of 25 known AI crawler tokens. Those names are self-identified, and section 1.2 is the reason not to trust one on its own. Here it does not weaken the result: a forged name can only add rows to the crawler column, which widens the gap below rather than closing it. Counting requests for three files that a crawler might plausibly want - the rules, the map, and the summary written for it:

crawler                requests  robots.txt  sitemap  llms.txt
Bytespider                 9031          75        0         0
ChatGPT-User               6729           0        4         0
ClaudeBot                  6665         978     1824         0
Amazonbot                  5442           2       16         0
PerplexityBot              3711         340       25         0
Meta-ExternalAgent         3505           0       17         0
Applebot                   2592         149        5         0
GPTBot                     1568           0      288         0
OAI-SearchBot              1506         406        2         0
CCBot                       705          27       17         1
11 further tokens          2166         432        2         0
--------------------------------------------------------------
all AI crawlers           43620        2409     2200         1

One request in 73 days. It came from CCBot/2.0 on 20 July 2026, from an AWS address whose reverse lookup returns NXDOMAIN, so by the standard of section 6.3 it cannot be confirmed as Common Crawl at all. Meanwhile the same crawlers asked for robots.txt 2,409 times and for a sitemap 2,200 times. They are not indifferent to files at the root of a site. They fetch the two that are standardised and ignore the one that is not.

Strip the AI crawlers out and 43 other requests for /llms.txt remain in the same period. Eleven are SEO and data-broker tools, seven are bots that exist because of the file itself - one of them announces "llms.txt corpus collection" in its user agent - six are performance tools loading it as if it were a page, two are vulnerability scanners, and the remainder carry ordinary browser user agents. The measurable audience for an llms.txt is people auditing llms.txt files.

The cost of adding one is ten minutes, and it does no harm. Just do not spend a week on it, and do not let it substitute for having the words in your HTML, which is the thing every crawler in this article definitely does read.

7.5 Where Crawler Control Stops

Every control in this article governs one thing: whether a machine may fetch a page today. None of them reaches backwards.

A model trained last year holds what your pages said last year. Disallowing a training crawler today changes the next model, not the one answering questions now.
A search index holds a copy taken when it last crawled. Blocking the crawler stops the refresh; it does not empty the index. That is what the removal tools in the search consoles are for, and there is no equivalent for a model. On the Google side, Search Console is also where you see which of your pages are indexed at all.
Your content also arrives through other doors: a scraped copy on another domain, a syndication partner, a quotation in a forum, an old page in a public archive. A perfect robots.txt does nothing about any of them.

This is the honest limit of the subject. You are choosing what happens next, on your own server, to visitors who identify themselves. It is worth doing, and it is not the same as control.

8. Best Practices

Decide per job, not per company. Training, search indexing and on-demand fetching deserve three separate answers. Write the file that says all three.
Put the words in the first response. Anything that arrives only after JavaScript may reach a browser and reach nothing else. Server-render the content you want quoted, and use the word count test of section 4.2 to prove it.
Copy your generic rules into every per-bot group. A specific group replaces the * group; it does not add to it.
Keep robots.txt boring and reachable. Root of the host, 200, text/plain, small. Watch it during deploys: a 5xx means complete disallow for anyone who reads it in that window.
Never treat robots.txt as access control. The user-initiated fetchers say they may ignore it, and nothing stops anyone else from doing the same. Private means authenticated.
Verify before you act on a name. Reverse DNS with a forward confirmation for Google, Apple and Microsoft; published address lists for the AI vendors; and log which creationTime you matched against.
Measure in your own log. Which tokens arrive, how often, which URLs, and whether they ever fetch an asset. That is data about your site, and it beats every general claim, including the ones in this article.
Check what the edge does. If a CDN sits in front of you, its bot rules and your file are two answers to one question. Test from outside your own network.
Re-read the vendor pages twice a year. Tokens get added, renamed and retired, and rules against retired names do nothing. The authoritative pages are: OpenAI, Anthropic, Perplexity, Google, Apple, Meta and Common Crawl, with RFC 9309 as the rulebook underneath all of them.

9. Common Mistakes

Myth	Reality
"I blocked `GPTBot`, so my content stays out of ChatGPT."	`GPTBot` collects training data. `OAI-SearchBot` builds the search index and `ChatGPT-User` fetches pages live. Three tokens, three decisions.
"`robots.txt` stops an AI from reading the page."	It stops the crawlers that honour it. The user-initiated fetchers document that they may not, and it was never access control for anyone else either.
"I disallowed `Google-Extended` but it still appears in my log."	It cannot appear. It sends no requests and has no user-agent string of its own; it only tells Google what may be done with what Googlebot already fetched.
"A crawler sees what I see."	One request, no cookies, no session, no language header, and for most AI crawlers no rendering. What arrives is your first response, nothing more.
"Blocking training crawlers removes my content from the models."	It affects future crawls and future models. Nothing you write in a text file today reaches a model trained last year.
"The user agent tells me who it is."	It is free text chosen by the client. Every `GPTBot` log line in section 1.2 was produced by `curl`. Verify with reverse DNS or a published address list.
"An `llms.txt` gets me cited."	No crawler documentation from any major operator says the crawler reads one. Five of 21 sampled sites publish one, all of them developer platforms. In 73 days of my own logs, 43,620 AI crawler requests produced a single fetch of the file, and that one could not be verified.
"No `robots.txt` means bots stay away."	The opposite. RFC 9309: a `4xx` means "the crawler MAY access any resources on the server". It is a `5xx` that means complete disallow.

And the traps that cost real time:

Rules against retired tokens. anthropic-ai and Claude-Web are dead names. So is any rule you copied from an article written before the vendor split one bot into three.
Disallow plus a meta directive. A page you disallow is never fetched, so the noarchive or noindex inside it is never read. Pick one mechanism per outcome.
A per-bot group that quietly grants everything. Section 5.3. Copy your generic Disallow lines into every specific group you write.
Blocking by IP address. The ranges sit in cloud providers and move. Worse, if your block also catches the request for robots.txt, the crawler cannot read your rules at all, and under RFC 9309 an error there has consequences of its own.
Blocking CSS and JavaScript paths. A renderer that cannot fetch your stylesheet may treat content as hidden. This has been bad advice for Googlebot for a decade and it has not improved.
Trusting a vendor address list without checking its age. One of the files in section 6.3 has not been regenerated since January 2024. Log the creationTime you matched against.
Serving bots a different page. The cure for a JavaScript-only page is server rendering or prerendering for everyone, not a special version for machines. Cloaking is detectable, it breaks the moment a user-agent string changes, and it makes every future debugging session twice as hard.
Infinite URL spaces. Faceted filters, sort parameters and calendar components generate endless valid URLs: /events/2026/09 has a next link, and so does /events/2197/04. Crawlers have fought this for decades, and the AI ones inherit it along with Cloudflare's observation that most AI crawl traffic re-fetches pages that did not change. Cap the combinations, canonicalise, and keep the generated space out of the sitemap.
Judging AI traffic by referrals alone. Most of the value of being read is a citation that never becomes a click. Count the fetches too, and compare them to what the answers actually say about you.

10. Summary

"AI crawler" covers three jobs: training, search indexing, and fetching one page because a person just asked. They arrive under different names and answer to different rules.
A fetch is one HTTP request with almost no headers, no cookies and no session, and for most AI crawlers no rendering. Your first response is the whole conversation.
The -User fetchers state in their own documentation that they may ignore robots.txt, because a person asked for the page. Private content needs authentication.
Google-Extended and Applebot-Extended are not crawlers. They are usage switches that never send a request and never appear in a log.
A specific User-agent group replaces the * group instead of adding to it, which turns a friendly per-bot Allow into an accidental grant.
A user-agent string is a claim. Reverse DNS with a forward confirmation proves Google, Apple and Microsoft; the AI vendors run in cloud address space and publish JSON address lists instead, of very uneven freshness.
The mechanism is moving on: signed requests instead of guessed identity, 402 with a price header instead of a yes or no, and an IETF vocabulary that states what may be done rather than who may knock.

THE THREE JOBS, AND THE TOKENS THAT DO THEM
  training      GPTBot  ClaudeBot  CCBot  Meta-ExternalAgent
  search index  OAI-SearchBot  Claude-SearchBot  PerplexityBot  Bingbot
  on demand     ChatGPT-User  Claude-User  Perplexity-User  Meta-ExternalFetcher
  usage only    Google-Extended  Applebot-Extended   (never send a request)

A ROBOTS.TXT THAT SAYS ALL THREE THINGS
  User-agent: GPTBot          User-agent: OAI-SearchBot
  Disallow: /                 Allow: /
  # repeat per training bot   # repeat per search bot
  # and copy your generic Disallow lines into every specific group

SEE WHAT THEY SEE
  curl -sS -A "GPTBot" https://example.com/page -o p.html -w "%{http_code}\n"
  sed 's/<[^>]*>/ /g' p.html | wc -w        # words in the first response
  document.body.innerText.split(/\s+/).length  # words after rendering

FIND THEM IN THE LOG
  grep -oiE 'GPTBot|ClaudeBot|PerplexityBot|CCBot|ChatGPT-User' access.log \
    | sort | uniq -c | sort -rn
  # then: did that IP fetch any .css .js or image? if not, it did not render

PROVE WHO CALLED
  dig +short -x 66.249.66.1        → crawl-66-249-66-1.googlebot.com
  dig +short crawl-66-249-66-1.googlebot.com   → 66.249.66.1   (must match)
  openai.com/gptbot.json  openai.com/searchbot.json  openai.com/chatgpt-user.json
  claude.com/crawling/bots.json    www.perplexity.com/perplexitybot.json
  search.developer.apple.com/applebot.json  index.commoncrawl.org/ccbot.json
  developers.google.com/static/crawling/ipranges/common-crawlers.json
  www.bing.com/toolbox/bingbot.json

RFC 9309 RULES WORTH REMEMBERING
  4xx on robots.txt  → crawler MAY access anything
  5xx on robots.txt  → crawler MUST assume complete disallow
  cache             → SHOULD NOT exceed 24 hours
  your own header   → serve robots.txt with expires 1d; a blanket static-file 30d delays every new rule
  group matching    → case-insensitive token; most specific group wins, never merged with *
  path matching     → case sensitive; longest match wins; allow wins a tie
  parsing limit     → at least 500 KiB
  # all figures and files verified 8 September 2026; server-log figures 15 September 2026

Every number, quotation and file in this article was produced on 8 September 2026 from primary sources: the crawler documentation of OpenAI, Anthropic, Perplexity, Google, Apple, Meta and Common Crawl; Google's documentation on AI features and your website; RFC 9309, read in full rather than summarised, and the AIPREF vocabulary and attachment drafts; Microsoft's Bing Webmaster Blog post of September 2023; Cloudflare's blog post of 1 July 2025 and its 2026 press release; and the 1994 Standard for Robot Exclusion. The address-list counts, the robots.txt files of the eight sampled publishers, the llms.txt scan of 21 sites, the DNS lookups and the verification script output were all run that day. The two-request comparison in section 1.2 came from a page built for the purpose, fetched once with curl and once with Playwright in headless Chromium. The server-log measurement in section 7.4 was taken on 15 September 2026 from this site's own Apache access logs covering 5 July to 15 September 2026, with requests originating from my own addresses excluded. Section 7.2 was revised on 16 September 2026 from Cloudflare's blog post of 15 September 2026 on accountable mixed-use crawlers, its press release of the same date, its migration notice emailed to account holders on 16 September 2026, and the Content Signals Policy as served at contentsignals.org that day.

And when an assistant keeps telling people you close at five, months after you changed the sign, the page it is reading is usually still sitting on your server: the words a browser paints and the words a fetcher receives were never the same page.

Need help with a web or technology problem?

Prefer to solve it together instead of spending hours searching for answers?

With db8 Website Support, I offer professional paid help with websites, internet services and technology-related questions and problems. We work together in a live online session, so you can ask questions, make decisions immediately, see exactly what I do, and learn along the way.

Get web & technology support

Peter Martin

Joomla Specialist

Peter is a Joomla specialist and a Linux admin for fast, secure and scalable websites.

Help with the web?

or technology related problems?

Stuck with a website, internet or technology problem? Get professional help in a live online session, solve the problem together and learn along the way.

Contact db8!