Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Google: 8 rows

In short

Google has 8 rows in the Index: Googlebot, Google-Extended, Google-CloudVertexBot, GoogleOther, Storebot-Google, Google-Agent, Gemini in Chrome and Google-GeminiNotebook. They cover 5 purposes: search index, training, other, task agent, live answer fetch. 5 of them have a robots.txt token and 2 publish nothing a site can verify.

Google: 8 rows.

Operator Token Purpose Respects robots.txt Verification Block rate 2026-11-04 Profile
Google Googlebot Search index Yes Reverse DNS 2.8% Profile
Google Google-Extended Training Yes None published 16.0% Profile
Google Google-CloudVertexBot Other Yes Reverse DNS 10.9% Profile
Google GoogleOther Other Yes Reverse DNS not measured Profile
Google Storebot-Google Search index Yes Reverse DNS not measured Profile
Google Google-Agent no token Task agent May not apply Web Bot Auth no token Profile
Google Gemini in Chrome (auto browse) no token Task agent No docs None published no token Profile
Google Google-GeminiNotebook (was Google-NotebookLM) no token Live answer fetch May not apply IP list no token Profile

Vendor profile: Google, with its products, standards, our probes and what it does not do.

Verification Ledger entry

Summary
Five IP lists, reverse DNS for every class, and an experimental key directory for Google-Agent. Nothing for auto browse in the user's Chrome.
IP lists
Reverse DNS
crawl-*.googlebot.com, geo-crawl-*.geo.googlebot.com, rate-limited-proxy-*.google.com, *.gae.googleusercontent.com, google-proxy-*.google.com
Key directory
https://agent.bot.goog/.well-known/http-message-signatures-directory (Google-Agent, subset of requests, experimental)
Cloudflare
not checked
Visa and Mastercard
Visa TAP: no public operator-level registration found (checked 2026-11-04). Mastercard Agent Pay: no public operator-level registration found (checked 2026-11-04).

Block rates for Google tokens

TokenRefused, top 10,000NamedVia *All crawled
Google-Extended16.0%12.6%3.4%12.1%
Google-CloudVertexBot10.9%7.1%3.8%8.3%
Googlebot2.8%0.5%2.3%2.1%

Based on 5,938 top-10,000 sites with a parsed robots.txt; crawl of 2026-11-04.

In the news

Sources

Last reviewed 2026-11-04