Is a robot really who it claims to be? Reverse DNS and published IP ranges

A user-agent is a string that any HTTP client can declare. A script presenting itself as "Googlebot" or "GPTBot" is neither, as far as we are concerned, until it has been checked. The question cuts both ways: not blocking a real search robot in the belief you are blocking an imitator, and not letting through a scraper borrowing a respectable name. The major robot operators all publish a verification method, and those methods differ.

Two families of verification

The first method is reverse DNS: take the request's IP address, ask DNS for the associated host name, check that the name belongs to a domain the operator has announced, then resolve that name back to an IP address and check you land on the address you started from. That double resolution stops a third party from claiming a domain it does not control, even where it controls the reverse DNS of its own address.

The second method is the published address list: the operator maintains a file, usually JSON, holding the IP ranges its robots come from. You simply check that the request's address falls inside one of those ranges. It is quicker to automate and does not depend on DNS, but the list has to be refreshed regularly.

Who publishes what

Nearly all these files follow the same schema, popularized by Google: an array of IPv4 and IPv6 prefixes with a creation date. A script that reads one reads the others. The creation date is worth reading: it says when the list was last produced. A list several months old is not wrong for that (the major operators' ranges move slowly), but it justifies a reverse DNS check whenever an address is missing and you have doubts.

The traps you hit in practice

1. The right URL, the wrong answer. Some documentation pages answer 200 with HTML when you expected JSON, because the address changed or a redirect leads to a home page. A script testing only the HTTP code will believe it holds a current list. Check the content type and the presence of the prefix array.
2. IPv6 is not published everywhere. Some operators publish IPv4 ranges only. A request arriving from an IPv6 address with the right user-agent cannot then be confirmed against the list, which does not make it spoofed. Plan for that case rather than blocking by default.
3. The firewall upstream. If a CDN or an application firewall filters traffic before the server, the address in the logs can be the CDN's rather than the robot's, unless a header carries the original IP. The major CDNs set one by default, but an intermediate layer can overwrite it. We come back to this in the article on the blind spot of edge blocking.

The method, step by step

To check a suspect request by hand:

1. Take the IP address and the user-agent from the log.
2. If the operator publishes a list, download the day's JSON and test whether the address belongs to one of the prefixes. An address outside the list with an operator's user-agent is spoofed, except in the uncovered IPv6 case.
3. If the operator uses reverse DNS, resolve the address to a host name, check the domain, then resolve that name back to an address and compare. Both resolutions have to agree.
4. Record the result. A log of detected spoofing helps tune the firewall and helps you justify yourself the day a robot operator disputes a block.

To automate it, refresh the lists daily, keep the previous version in case a download fails, and never block an address missing from a list without a reverse DNS check first: ranges move, and a genuine robot arriving from a new range would be taken for an impostor.

What these lists also say about roles

OpenAI publishes one list per robot: one for OAI-SearchBot, one for ChatGPT-User, one for GPTBot. That separation is not an organizational detail. It lets you know, address by address, whether a request came from search indexing, from a page read on a user's behalf inside a conversation, or from collection for training. Three purposes a publisher may want to treat differently, and which a plain filter on the word "GPT" merges into one. Anthropic groups its robots in a single file but draws the same role distinction inside it. A crawl log keeping only the robot's name loses that information; a log keeping the address lets you reconstruct it afterwards.

What to do with an impostor

A client pretending to be Googlebot is not necessarily malicious: SEO tools, monitoring probes and internal scripts do it out of convenience. But it consumes resources and distorts crawl statistics. You can let it through while counting it separately in reports, rate-limit it, or block it at the firewall. The choice depends on what the site publishes and on the load you observe. A media property selling information has reasons to block; one that lives on its visibility has reasons to let through and measure. What you should not do is block an entire signature, anything containing "bot" for example, at the risk of shutting out the very robots you wanted to welcome.

Why a media property has to care

A brochure site can ignore the question. A media property aiming to be cited by generative engines cannot. The first reason is defensive: AI robot spoofing is common, and a badly tuned block aimed at it usually catches the real ones too. The second is analytical: measuring what AI tools actually do with a site means knowing, request by request, which robot came, in which role, and whether it was genuine. Without that check, a curve of "AI robot visits" counts the imitators too. The log reading that makes the measurement possible is described in "Reading an access log in 30 minutes", and the list of the robots themselves in "robots.txt and AI".