Reading an access log in 30 minutes: what AI robots really do on your site

No dashboard tells a publisher whether ChatGPT, Claude or Perplexity read its pages. Search Console covers Google only, and generative engines supply nothing. The one source that exists is the server's request log, the access log, which every host produces and almost nobody opens. In half an hour, with the tools present on any Linux server, you can read from it who comes, what for, and what the server answered. Here is the method we apply, step by step, with the commands. How the half hour breaks down: five minutes to extract, five to count by robot, ten to check the addresses, five for the response codes, five for the paths and the cadence.

Every command below assumes a combined-format log and runs on the file extracted at step 1. They use only grep, awk, sort and uniq.

What a line contains

An access log in the combined format, the Apache and nginx default, holds one line per request. Each line gives, in order: the client IP address, the date and time, the method and the path requested, the response code, the size sent, the referer and the user-agent. For robot analysis, four fields count:

First check, before any analysis: is the IP address recorded the visitor's, or an intermediary's? Behind a CDN or a proxy, the server sees the intermediary's address, unless it is configured to read the header carrying the original one. If every request seems to come from a few dozen addresses, the verification step described below will not be possible, and the configuration has to be fixed before going further. We described that trap in "What the firewall hides from your logs".

Step 1: isolate the period and the AI robots

You work on a complete period, seven days at least, because AI robots do not come every day and conclusions drawn from twenty-four hours are almost always wrong. You then extract the lines whose user-agent contains one of the tokens published by the providers: GPTBot, ChatGPT-User, OAI-SearchBot for OpenAI; ClaudeBot, Claude-User, Claude-SearchBot for Anthropic; PerplexityBot and Perplexity-User for Perplexity; Google-Extended does not appear in the logs, since it is a robots.txt token and not a robot. The reference lists are those of OpenAI, Anthropic and Perplexity.

A grep filter on those tokens, piped into a working file, is enough. On a mid-sized media property, that yields a few hundred to a few thousand lines a week.

zcat -f access.log* \
  | grep -E 'GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User' \
  > ia.log

Step 2: count by robot and by role

Every provider runs several robots that do different jobs, and that distinction is the most useful one in the whole analysis. A training robot collects pages for future models. A search robot builds an index to answer questions. An on-demand robot goes and fetches a page for a person putting a question to the assistant. We set out the roles and the tokens in "robots.txt and AI".

So you count the lines by user-agent, and group them into three columns: training, search, user agent. The result reads as follows:

• Training requests on their own mean the site is being collected, though nothing says it is cited.
• Search requests indicate the site is indexed by the generative engine, a necessary condition for being cited.
• User agent requests are the most valuable signal: each one matches a user question for which the assistant went and fetched that page.

grep -oE 'GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User' ia.log \
  | sort | uniq -c | sort -rn

Step 3: check that the robots are authentic

A user-agent is declared freely. Any script can present itself as GPTBot. Before counting, you check that the IP addresses belong to the ranges published by each provider: OpenAI, Anthropic and Perplexity publish range files, and Google documents verifying its robots through reverse DNS. You extract the distinct addresses per user-agent, compare them with the ranges, and set aside anything that is not there. The complete method, with its traps, is in "Is a robot really who it claims to be?".

We find some on most of the sites we audit: requests declaring themselves Googlebot or GPTBot from addresses belonging to neither Google nor OpenAI. We have no publishable volume figure to attach to that; the test, on the other hand, takes one command. Counting without checking means risking measuring scrapers in disguise.

# distinct addresses for one robot, to compare with the published ranges
awk '/GPTBot/ {print $1}' ia.log | sort -u

Step 4: read the response codes

For each authentic robot, you break the response codes down. The table expected on a healthy site is almost entirely 200s, with a few 304s if the server handles conditional requests, and a few 404s on withdrawn URLs. Three anomalies deserve action:

A reminder about refusals: a 403 returned by the origin server appears in the log. A 403 returned by the CDN does not, since the request never arrived. A log with no error at all is therefore no proof that no request is being refused.

# the response code is the 9th field of the combined format
awk '/GPTBot/ {print $9}' ia.log | sort | uniq -c | sort -rn

Step 5: look at which pages are read

Many publishers stop before this step, and it is the one that teaches the most. For each search robot and each user agent, you count the paths requested and compare them with the list of published pages. That count answers questions no other tool settles:

1. Which pages are never read? An article published two months ago that no search robot has requested is probably not in the sitemap, or not reachable through a link.
2. Which pages do the user agents read? Those are the ones assistants really cite. Comparing them with the pages you would like to see cited says where to put the writing effort.
3. Does the robot come back to modified pages? If a page updated last week has not been requested again, the freshness signal is not getting through. We explained why in "A sitemap with identical dates is useless".

# the most-read paths for the search robot, 7th field
awk '/OAI-SearchBot/ {print $7}' ia.log | sort | uniq -c | sort -rn | head -30

Step 6: note the cadence and take the measurement again

The last reading is about time. For each robot, you record the number of visit days over the period, the interval between two passes, and the usual time of day. A search robot that comes every day has folded the site into its cycle; a robot that came once to read everything and has since come back only for the sitemap is waiting for a change signal. That cadence is the reference the following measurements will be compared against, once a week or once a month depending on the publishing rhythm.

The working file has to be kept as well. Retention depends on the host and the contract, and it is often counted in days rather than months: check yours before you need it. Past that, the data no longer exists, and a question asked in December about what happened in September will stay unanswered.

# days of visit, then the usual hour
awk '/GPTBot/ {print substr($4,2,11)}' ia.log | sort -u
awk '/GPTBot/ {print substr($4,14,2)}' ia.log | sort | uniq -c

The three mistakes almost everyone makes the first time

The first is concluding from a sample: the last few hours, the first page of a tool, a single day. A robot may have read the whole site on a Tuesday evening and not show up again for eight days. You look at the entire period before saying it does not come.

The second is testing from your own machine. Sending a request with GPTBot's user-agent from an office network and getting a 403 proves only that impostors are refused; the real robot may well get through. The only test worth anything is the comparison between what the log shows for the authentic robot and what you get yourself.

The third is forgetting what never reaches the log. A site behind a CDN that refuses AI robots will have a perfectly clean access log, empty of any AI. Reading the logs starts with reading the CDN configuration, and with the CDN's own logs when they are accessible.

What that half hour buys you

At the end of the exercise, a publisher knows three things no dashboard will hand over: which generative engines index the site, which pages they read, and whether real people arrive through an assistant. That is the basis of any AI visibility strategy, ahead of the share of voice measurements we describe in "Measuring your visibility in AI answers". An editorial platform can automate that reading and display it every day; but the first time, you have to have done it by hand, to know what you are looking at.