Crawlers and logs

Blog sections

On a bright orange background, navy blue parallel bands shift vertically at irregular intervals, evoking the reading of logs and request rhythms, with a grainy texture and halftone dots reminiscent of a risograph print.

Crawlers and logs

GPTBot, ClaudeBot, PerplexityBot: reading your logs to see what AI really does with your site

AI bots are no longer marginal visitors: they crawl the web to collect, index or query content. In server logs they are spotted first...

September 11, 20265 min read

Irregular orange vertical bands, some seemingly interrupted or offset, cross a saturated blue-green background with a dot screen and heavy grain, evoking a sequence of analysed requests marked by anomalies in a log.

Crawlers and logs

Reading an access log in 30 minutes: what AI robots really do on your site

No dashboard tells a publisher whether ChatGPT, Claude or Perplexity read its pages. Search Console covers Google only, and generative engines supply nothing. The...

September 2, 20267 min read

On a dark blue background, a regular grid of white dots is abruptly interrupted at its centre by a wide orange band, evoking a clean break in a flow of data.

Crawlers and logs

What the firewall hides from your logs: the blind spot of edge blocking

A site owner opens the logs, searches for GPTBot, ClaudeBot or PerplexityBot, finds nothing, and concludes that AI tools take no interest in the...

August 17, 20265 min read

A network of white lines connected by yellow nodes on a midnight blue background, evoking cross-checking and rigorous control of connections, with a heavy printed grain texture.

Crawlers and logs

Is a robot really who it claims to be? Reverse DNS and published IP ranges

A user-agent is a string that any HTTP client can declare. A script presenting itself as "Googlebot" or "GPTBot" is neither, as far as...

July 24, 20265 min read

Three wide parallel bands, offset and segmented, fill a deep navy blue background, with grainy textures and print flaws that evoke several distinct filters or roles layered on top of one another.

Crawlers and logs

robots.txt and AI: blocking GPTBot does not block ChatGPT

In 2023, when OpenAI published the name of its training robot, thousands of sites added two lines to their robots.txt: User-agent: GPTBot then Disallow...

July 6, 20265 min read