---
title: "How to read agent traffic in your logs | Agent Ready Crawlers"
description: "How to read AI crawler traffic in your server logs: match the token, check the source address against the operator's list, and log Signature-Agent."
canonical: https://agentreadycrawlers.com/guides/read-agent-traffic-in-logs
site: "Agent Ready Crawlers"
last_updated: 2026-11-04
date: 2026-11-04
section: "Guides"
---

# How to read agent traffic in your logs | Agent Ready Crawlers

> How to read AI crawler traffic in your server logs: match the token, check the source address against the operator's list, and log Signature-Agent.

Source: https://agentreadycrawlers.com/guides/read-agent-traffic-in-logs
Site: Agent Ready Crawlers
Last reviewed: 2026-11-04

# How to read agent traffic in your logs

In short

Find the token, check the source address against the operator's published list or reverse DNS, and log the Signature-Agent header so signed agents are visible. OpenAI publishes one JSON file of IP prefixes per crawler: gptbot.json, searchbot.json, chatgpt-user.json and adsbot.json. Google verification is reverse DNS to googlebot.com, google.com or googleusercontent.com, then a forward lookup that must return the same IP.

A user-agent string is a claim. A log line becomes evidence when the token, the source address and, for signed agents, the Signature-Agent header agree.

## In practice

1.  Log the user agent, the request path, the source address and the Signature-Agent header on every request. Log paths because OpenAI and Google both note that a robots.txt fetch can carry a marker or a different string.
2.  Match the token in the user agent against the [Crawler Index](https://agentreadycrawlers.com/crawlers/). A match on the string alone is unverified.
3.  Check the source address. For OpenAI, compare it with the JSON file for that crawler (gptbot.json, searchbot.json, chatgpt-user.json or adsbot.json). For Google, resolve the address to googlebot.com, google.com or googleusercontent.com, then resolve that name forward and require the same address.
4.  Treat copies as likely. Common Crawl warns that other crawlers falsely identify as CCBot and recommends reverse DNS to `*.crawl.commoncrawl.org`.
5.  Expect some agents to look like ordinary visitors. Claude in Chrome, Comet and Gemini auto browse send the user's own user agent and IP, so a log cannot single them out.
6.  For signed agents such as ChatGPT's Cloud browser, keep all three headers (Signature, Signature-Input, Signature-Agent) through every proxy, or verification fails downstream.

## What the sources say

-   Primary source OpenAI publishes one JSON file of IP prefixes per crawler: gptbot.json, searchbot.json, chatgpt-user.json and adsbot.json. [Source](https://platform.openai.com/docs/bots), read 2026-11-04.
-   Primary source OpenAI and Google both note that robots.txt fetches can carry a marker or a different string, so log paths alongside user agents. [Source](https://platform.openai.com/docs/bots), read 2026-11-04.
-   Primary source Google verification is reverse DNS to googlebot.com, google.com or googleusercontent.com, then a forward lookup that must return the same IP. [Source](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests), read 2026-11-04.
-   Primary source ChatGPT's Cloud browser sends Signature, Signature-Input and Signature-Agent: "https://chatgpt.com"; intermediate proxies must preserve all three headers. [Source](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting), read 2026-11-04.
-   Secondary source Agents that run in the user's own browser (Claude in Chrome, Comet, Gemini auto browse) send the user's user agent and IP, so logs show them as ordinary visitors. [Source](https://www.centinelanalytica.com/blog/ai-browser-agents), read 2026-11-04.
-   Primary source Common Crawl warns that other crawlers falsely identify as CCBot and recommends verifying by reverse DNS to \*.crawl.commoncrawl.org. [Source](https://commoncrawl.org/ccbot), read 2026-11-04.

## Related

-   [The Verification Ledger](https://agentreadycrawlers.com/verification/)
-   [GPTBot profile](https://agentreadycrawlers.com/crawlers/gptbot/)
-   [CCBot profile](https://agentreadycrawlers.com/crawlers/ccbot/)
-   [ChatGPT agent profile](https://agentreadycrawlers.com/crawlers/chatgpt-agent/)
-   [Reverse DNS verification](https://agentreadycrawlers.com/glossary/reverse-dns-verification/)
-   [How to test whether AI agents can reach your site](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/)
-   [The names sites write in robots.txt](https://agentreadycrawlers.com/crawlers/named-in-robots-txt/)

## Sources

-   [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04
-   [Verify requests from Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) read 2026-11-04
-   [ChatGPT Work's Cloud browser allowlisting](https://help.openai.com/en/articles/11845367-chatgpt-agent-allowlisting) read 2026-11-04
-   [centinelanalytica.com: ai browser agents](https://www.centinelanalytica.com/blog/ai-browser-agents) read 2026-11-04
-   [CCBot](https://commoncrawl.org/ccbot) read 2026-11-04

Last reviewed 2026-11-04