Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

How to read agent traffic in your logs

In short

Find the token, check the source address against the operator's published list or reverse DNS, and log the Signature-Agent header so signed agents are visible. OpenAI publishes one JSON file of IP prefixes per crawler: gptbot.json, searchbot.json, chatgpt-user.json and adsbot.json. Google verification is reverse DNS to googlebot.com, google.com or googleusercontent.com, then a forward lookup that must return the same IP.

A user-agent string is a claim. A log line becomes evidence when the token, the source address and, for signed agents, the Signature-Agent header agree.

In practice

  1. Log the user agent, the request path, the source address and the Signature-Agent header on every request. Log paths because OpenAI and Google both note that a robots.txt fetch can carry a marker or a different string.
  2. Match the token in the user agent against the Crawler Index. A match on the string alone is unverified.
  3. Check the source address. For OpenAI, compare it with the JSON file for that crawler (gptbot.json, searchbot.json, chatgpt-user.json or adsbot.json). For Google, resolve the address to googlebot.com, google.com or googleusercontent.com, then resolve that name forward and require the same address.
  4. Treat copies as likely. Common Crawl warns that other crawlers falsely identify as CCBot and recommends reverse DNS to *.crawl.commoncrawl.org.
  5. Expect some agents to look like ordinary visitors. Claude in Chrome, Comet and Gemini auto browse send the user's own user agent and IP, so a log cannot single them out.
  6. For signed agents such as ChatGPT's Cloud browser, keep all three headers (Signature, Signature-Input, Signature-Agent) through every proxy, or verification fails downstream.

What the sources say

Sources

Last reviewed 2026-11-04