Google: 8 rows
In short
Google has 8 rows in the Index: Googlebot, Google-Extended, Google-CloudVertexBot, GoogleOther, Storebot-Google, Google-Agent, Gemini in Chrome and Google-GeminiNotebook. They cover 5 purposes: search index, training, other, task agent, live answer fetch. 5 of them have a robots.txt token and 2 publish nothing a site can verify.
| Operator | Token | Block rate |
|---|---|---|
| Googlebot | 2.8% | |
| Google-Extended | 16.0% | |
| Google-CloudVertexBot | 10.9% | |
| GoogleOther | ||
| Storebot-Google | ||
| Google-Agent | ||
| Gemini in Chrome (auto browse) | ||
| Google-GeminiNotebook (was Google-NotebookLM) |
Vendor profile: Google, with its products, standards, our probes and what it does not do.
Verification Ledger entry
- Summary
- Five IP lists, reverse DNS for every class, and an experimental key directory for Google-Agent. Nothing for auto browse in the user's Chrome.
- IP lists
- developers.google.com/static/search/apis/ipranges/googlebot.json
- developers.google.com/static/search/apis/ipranges/special-crawlers.json
- developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json
- developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json
- developers.google.com/static/search/apis/ipranges/user-triggered-agents.json
- Reverse DNS
- crawl-*.googlebot.com, geo-crawl-*.geo.googlebot.com, rate-limited-proxy-*.google.com, *.gae.googleusercontent.com, google-proxy-*.google.com
- Key directory
- https://agent.bot.goog/.well-known/http-message-signatures-directory (Google-Agent, subset of requests, experimental)
- Cloudflare
- not checked
- Visa and Mastercard
- Visa TAP: no public operator-level registration found (checked 2026-11-04). Mastercard Agent Pay: no public operator-level registration found (checked 2026-11-04).
Block rates for Google tokens
| Token | Refused, top 10,000 | Named | Via * | All crawled |
|---|---|---|---|---|
| Google-Extended | 16.0% | 12.6% | 3.4% | 12.1% |
| Google-CloudVertexBot | 10.9% | 7.1% | 3.8% | 8.3% |
| Googlebot | 2.8% | 0.5% | 2.3% | 2.1% |
Based on 5,938 top-10,000 sites with a parsed robots.txt; crawl of 2026-11-04.
In the news
- 2026-09-15Cloudflare adds Disallow AI Training and the Accountable label
- 2026-08-19Google retires the Google-NotebookLM user agent
Sources
Last reviewed 2026-11-04