robots.txt and AI: blocking GPTBot does not block ChatGPT

In 2023, when OpenAI published the name of its training robot, thousands of sites added two lines to their robots.txt: User-agent: GPTBot then Disallow: /. Many have believed ever since that ChatGPT can no longer read their pages. That belief is wrong, and the confusion comes from something the AI operators document plainly: they do not send one robot but several, each with its own role and its own token. Blocking one does not block the others.

Several robots, several roles, at every operator

At OpenAI, according to its bots documentation:

• GPTBot collects pages that may be used to train the models. It is the only one whose blocking protects against use in training.
• OAI-SearchBot feeds the index behind ChatGPT's built-in search. Blocking it removes your pages from ChatGPT's search results, and from the citations that follow.
• ChatGPT-User fetches a specific page when a user asks for it in a conversation, by pasting a URL or asking a question that triggers a read. It does not crawl the web on its own.

Anthropic draws the same separation, described in its help page on robots: ClaudeBot for training, Claude-SearchBot for the search index, Claude-User for user-requested fetches. Perplexity documents PerplexityBot (index) and Perplexity-User (on-demand fetch) in its crawler documentation.

Google and Apple: a control token separate from the robot

Google works differently. Googlebot remains the single robot that crawls pages for search, and those same pages feed AI Overviews. Control over training use goes through a separate token, Google-Extended, which is not a robot but a directive Googlebot reads, as the list of Google's common crawlers explains. Google states that this token changes nothing about page indexing. Apple does the same with Applebot-Extended, a token documented in its Applebot support article, which controls training use without touching Applebot itself.

To that list add CCBot, the robot of Common Crawl, whose open archives have been raw material for many models. Blocking CCBot is the broadest lever against training by operators who run no robot of their own.

The token table, and what each block implies

What robots.txt can and cannot do

The protocol is now standardized by RFC 9309. Three of its rules explain most of the surprises:

1. The file is only read at the host root. It is the host and not the domain: a robots.txt served on example.com does not govern blog.example.com, which is a separate host. A robots.txt placed in a subdirectory, under a /blog/ path hosted elsewhere for instance, is never consulted. Its directives, sitemap declarations included, are invisible to every robot. On a media property mounted on a subpath of a main site, the main site's robots.txt is what counts.
2. The most specific group wins, and it replaces the generic group. A User-agent: GPTBot group containing only Disallow: / does not inherit the rules of the User-agent: * group. If the generic group barred facets or files, the named robot has access to them again. It is a common mistake when groups pile up robot by robot over the months.
3. The file is an instruction, not a barrier. The operators named here state that they honor it. An operator who does not declare its name escapes any named rule. Blocking for real means acting at the server or firewall level, with the side effects we describe in the article on the blind spot of edge blocking.

Which policy to choose

The decision is editorial before it is technical. Three positions hold up:

• Open everything. The site accepts training and citation. It is the default position for a media property chasing visibility, inside the models included.
• Refuse training, accept citation. You block GPTBot, ClaudeBot, CCBot and disallow Google-Extended and Applebot-Extended, while letting OAI-SearchBot, Claude-SearchBot, PerplexityBot and the "User" robots through. It is the position of a publisher who wants to be cited without feeding the models for free.
• Close everything. The site gives up visibility inside AI answers. Defensible for an intranet or paid content, not for a media property that wants to carry weight on a search results page or in a generated answer.

The second position is widespread among news publishers, and often badly applied: in practice, many barred anything containing "GPT" or "Claude" in one stroke, and so barred the search and reading robots too. The result is a site absent from generated answers, without anyone on the team having decided it.

Check rather than assume

The check takes two moves. First reread the robots.txt as served at the root, token by token, with the table above in front of you. Then open the server logs to see which robots actually come, how often, and which ones get a 403 or a 404. An AI robot that never appears in the logs is not necessarily absent: it may have been stopped upstream by a firewall, before reaching the server. That check is set out step by step in "Reading an access log in 30 minutes".

For the Google side, Search Console provides a robots.txt report showing the last version Googlebot read, its fetch date and any parsing errors; Google's robots.txt introduction describes how to use it. For the other operators there is no equivalent: the logs are what count, and a manual test with each robot's user-agent remains the surest way to check what the server actually answers.

A media property that wants to be cited cannot rely on a file written two years ago. Tokens change, robots multiply, and a line added as a precaution one day in 2023 can cost every trace of presence in AI answers in 2026.