Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Crawler operator

Google

In short

Google separates its fetchers into three classes, each with a reverse DNS mask and a JSON range file, five files in all. A request never carries Google-Extended, which exists purely as a control word in robots.txt for Gemini training and grounding. Google-Agent traffic is partly signed with Web Bot Auth, an experiment described on 13 October 2026.

Type
Crawler operator
Homepage
google.com
In one line
Google runs common crawlers such as Googlebot, special-case crawlers and user-triggered fetchers, verifiable by reverse DNS and five IP files, and is testing Web Bot Auth signatures on part of Google-Agent's traffic.
Last read
2026-11-05

What Google does for this layer

From the vendor's own pages

Google says its common crawlers, Googlebot among them, always obey robots.txt rules when crawling automatically. (developers.google.com) Google-Extended has no HTTP user agent of its own and works only as a robots.txt control token for Gemini model training and grounding; Google says it does not affect inclusion or ranking in Google Search. (developers.google.com) Google states it intends to support the AIPREF shared training preference signal, written "user-agent: __usage_ai-training", in early 2027. (developers.google.com, dated 2026-10-30) Google's verification page sorts its requesters into common crawlers, special-case crawlers and user-triggered fetchers, each with its own reverse DNS mask and IP file. (developers.google.com)

Google says special-case crawlers such as AdsBot may or may not respect robots.txt, and that user-triggered fetchers ignore robots.txt because a user asked for the fetch. (developers.google.com) Google calls its Web Bot Auth use experimental, signs only a subset of Google-Agent requests under the identity https://agent.bot.goog, and tells sites to keep IP and reverse DNS checks as a fallback. (developers.google.com, dated 2026-10-13) Signed Google-Agent requests carry a Signature-Agent header set to g="https://agent.bot.goog", with the Signature and Signature-Input headers labelled g. (developers.google.com) Google-Agent is the user agent for agents hosted on Google infrastructure that browse and act on user request, and it uses the ranges in user-triggered-agents.json. (developers.google.com)

Google's Gemini Notebook fetcher now sends the token Google-GeminiNotebook; the former Google-NotebookLM agent was supported until August 2026. (developers.google.com)

What our crawl saw

Google's common-crawlers.json listed 317 prefixes, 170 IPv4 and 147 IPv6, with a creationTime of 2026-11-04 when fetched on 2026-11-05. (developers.google.com, dated 2026-11-04) Google's user-triggered-agents.json, the range file for Google-Agent, listed 20 prefixes (12 IPv4, 8 IPv6) on 2026-11-05. (developers.google.com, dated 2026-11-04) agent.bot.goog served a key directory with five Ed25519 keys and the content type application/http-message-signatures-directory+json on 2026-11-05. (agent.bot.goog)

Products and plans

ProductWhat it doesPrice modelFromAs of
Googlebot Common crawler for Google Search, Images, Video, News and Discover. No price recorded
Google-Extended Robots.txt token, with no user agent string, that controls Gemini training and grounding use. No price recorded
Google-Agent User agent for Google-hosted agents acting on user request; some requests signed with Web Bot Auth. No price recorded
Google-CloudVertexBot Crawler for site-owner-requested crawls that build Vertex AI Agents. No price recorded
Storebot-Google Crawler for Google Shopping surfaces. No price recorded
Google-GeminiNotebook User-triggered fetcher for sources added to Gemini Notebook projects. No price recorded

Standards

StandardStatusEvidence
RFC 9309 (robots.txt) Partial Common crawlers always obey robots.txt for automatic crawls; special-case crawlers may not; user-triggered fetchers ignore it. (developers.google.com)
Web Bot Auth Partial Experimental; a subset of Google-Agent requests signed as https://agent.bot.goog. (developers.google.com)
Content Signals No statement Google names the AIPREF signal for early 2027; Content-Signals is not mentioned (read 2026-11-05).
Published IP list Supports Five published range files, including user-triggered-agents.json for Google-Agent. (developers.google.com)
Reverse DNS check Supports Masks under googlebot.com, google.com and googleusercontent.com with a forward-confirm step. (developers.google.com)
Cloudflare verified bots No statement Google's crawler pages do not mention Cloudflare's list (read 2026-11-05).

Our checks

CheckResultMethodDate
Homepage fetch HTTP 200 GET with the crawl user agent after a robots.txt check 2026-11-05
Web Bot Auth key directory agent.bot.goog: 200, application/http-message-signatures-directory+json, 5 keys GET /.well-known/http-message-signatures-directory with the crawl user agent 2026-11-05
common-crawlers.json 317 prefixes (170 IPv4, 147 IPv6), creationTime 2026-11-04 GET and count prefixes 2026-11-05
special-crawlers.json 272 prefixes (136 IPv4, 136 IPv6), creationTime 2026-11-04 GET and count prefixes 2026-11-05
user-triggered-agents.json 20 prefixes (12 IPv4, 8 IPv6), creationTime 2026-11-04 GET and count prefixes 2026-11-05
user-triggered-fetchers-google.json 504 prefixes (252 IPv4, 252 IPv6), creationTime 2026-11-04 GET and count prefixes 2026-11-05

What Google does not do

  • Google does not sign every Google-Agent request, so a missing signature does not prove a request is fake.
  • Google-Extended cannot be blocked at the server by user agent, because Google sends no Google-Extended string in requests.
  • Google's common crawler and user-triggered fetcher pages list no token for Gemini auto browse in Chrome (read 2026-11-05).

Reading notes

Google's robots.txt disallows our crawl token for www.google.com/.well-known/, so that path was not probed for a key directory; the Web Bot Auth directory on agent.bot.goog was read instead.

Sources

Last reviewed 2026-11-05