What the firewall hides from your logs: the blind spot of edge blocking

A site owner opens the logs, searches for GPTBot, ClaudeBot or PerplexityBot, finds nothing, and concludes that AI tools take no interest in the content. That conclusion is often wrong, for a mechanical reason: when a firewall or a CDN refuses a request, it stops at the edge of the network and never reaches the origin server. It therefore exists in no origin log. Silent logs do not mean the robots are absent; they can mean precisely the opposite.

How a request can disappear

On a site served behind a CDN such as Cloudflare, every request crosses several layers before it reaches the server. The application firewall, described in Cloudflare's WAF documentation, evaluates rules and can block, redirect or serve a challenge. A bot management module, detailed in the Bots documentation, classifies automated traffic and applies a policy to it. Since 2024 these platforms have offered one-click blocking of AI robots, as Cloudflare announced in its post on blocking AI bots, and many sites switched the option on without measuring what it did.

Any request stopped at that level gets an answer manufactured by the CDN, usually a 403 or a challenge page. The origin server sees none of it. Apache, nginx and application logs do not contain it, and a log analysis tool that reads only the origin cannot count it.

The three wrong conclusions this produces

1. "No AI ever comes to us." The most common one. The site is in fact visited, but every visit is refused upstream. A rule switched on as a precaution, sometimes by a supplier, sometimes by a checkbox in an interface, quietly cuts off any chance of being cited.
2. "Our crawl success rate is 100%." A dashboard counting the 200 responses served by the origin shows a perfect score, since only accepted requests get there. The real rate, blocks included, can be very low.
3. "The robot ignores our robots.txt." If robots.txt is served by the origin but the robot's first request is blocked at the edge, it never read the file. The robot gets blamed for behavior whose cause is local.

How to check what happens at the edge

The checks, from simplest to most complete:

• Reread the CDN configuration. WAF rules, bot management mode, "block AI bots" options, refused user-agent lists. Read it as it stands, not as you believe you set it: on a site we audited this summer, a blocking rule inherited from an earlier configuration was cutting off robots nobody on the team thought they were blocking.
• Test with the real user-agents. Send a request yourself to a public page declaring each AI robot's user-agent, from outside the network, and look at the response code and who issued it. A 403 signed by the CDN while the origin answers 200 is direct proof of edge blocking.
• Read the CDN logs, not only the origin's. The large CDNs expose traffic statistics that include refused requests, by rule and by user-agent, sometimes by path. That is where the robots you thought absent show up. On Cloudflare this data is available through the interface and through the API with a token holding the Zone Analytics read permission; on a free plan, the breakdown by path requires the API.

A measured case

On another site, an industrial one audited the same summer, the origin logs showed no AI robot requests over several weeks, and the crawl dashboard displayed a perfect success rate. The CDN logs told a different story: several hundred AI robot requests refused with a 403 by a firewall rule, with no trace anywhere else. The initial conclusion, "the AIs do not come", was the exact reverse of reality: they came, regularly, and left without reading anything. The site appeared in no generated answer, not because engines were uninterested, but because of a security setting. The diagnosis took an hour; it might never have been made.

The IP address trap

A second CDN effect complicates reading the origin logs: the IP address the server sees is the CDN's, not the visitor's, unless the server is configured to read the header carrying the original address. When that setting is missing, every visitor, human and robot alike, appears to come from a few dozen addresses belonging to the CDN. The authenticity checks described in our article on reverse DNS and published IP ranges become impossible, and a genuine robot can be taken for an impostor, or the other way round. A proprietary header added by the CDN generally survives every intermediate hop where a generic header can be overwritten. At Cloudflare it is CF-Connecting-IP; the generic header, X-Forwarded-For, can be rewritten by any intermediary. Ask your host for the exact name of the one it adds, and check that it actually reaches the origin log.

Which policy to hold

The underlying question is editorial: a media property that wants to be cited by AI tools has no reason to block their search and reading robots at the edge. Blocking training is a legitimate decision, but it belongs in robots.txt, user-agent by user-agent, as we explain in "robots.txt and AI", not in a switch that refuses anything resembling an AI. Edge blocking only makes sense for proven abusive traffic: abnormal volumes, identity spoofing, scrapers that ignore robots.txt.

One precaution: do not block impostors with a broad user-agent rule either. A rule refusing any "GPTBot" coming from an address outside OpenAI's published ranges looks prudent; in practice it also refuses legitimate requests the day OpenAI adds a range your local list does not have yet. The right answer to spoofing is measurement and rate limiting, not blind refusal.

What measuring means once the block is lifted

When the edge lets traffic through, origin logs become a reliable source again, provided they carry the original address and keep the full user-agent. You can then count, by robot and by purpose (training, search, on-demand reading), what the AI tools actually read: which pages, how often, with which response code. That reading, rather than an origin dashboard, is what lets you state that content is reached by generative engines. We set out the method in "Reading an access log in 30 minutes".

The principle holds: an origin log proves only what reached the origin. Whatever was refused before leaves no trace there, and that is exactly what you have to go and find elsewhere before concluding anything about AI.