---
title: "Google crawlers and agents | Agent Ready Crawlers"
description: "Google in the Crawler Index: 8 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04."
canonical: https://agentreadycrawlers.com/crawlers/operator/google
site: "Agent Ready Crawlers"
last_updated: 2026-11-04
section: "Crawlers"
---

# Google crawlers and agents | Agent Ready Crawlers

> Google in the Crawler Index: 8 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04.

Source: https://agentreadycrawlers.com/crawlers/operator/google
Site: Agent Ready Crawlers
Last reviewed: 2026-11-04

# Google: 8 rows

In short

Google has 8 rows in the Index: Googlebot, Google-Extended, Google-CloudVertexBot, GoogleOther, Storebot-Google, Google-Agent, Gemini in Chrome and Google-GeminiNotebook. They cover 5 purposes: search index, training, other, task agent, live answer fetch. 5 of them have a robots.txt token and 2 publish nothing a site can verify.

Google: 8 rows.

| Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile |
| --- | --- | --- | --- | --- | --- | --- |
| Google | Googlebot | Search index | Yes | Reverse DNS | 2.8% | Profile |
| Google | Google-Extended | Training | Yes | None published | 16.0% | Profile |
| Google | Google-CloudVertexBot | Other | Yes | Reverse DNS | 10.9% | Profile |
| Google | GoogleOther | Other | Yes | Reverse DNS | not measured | Profile |
| Google | Storebot-Google | Search index | Yes | Reverse DNS | not measured | Profile |
| Google | Google-Agent no token | Task agent | May not apply | Web Bot Auth | no token | Profile |
| Google | Gemini in Chrome (auto browse) no token | Task agent | No docs | None published | no token | Profile |
| Google | Google-GeminiNotebook (was Google-NotebookLM) no token | Live answer fetch | May not apply | IP list | no token | Profile |

Vendor profile: [Google](https://agentreadycrawlers.com/vendors/google/), with its products, standards, our probes and what it does not do.

## Verification Ledger entry

Summary

Five IP lists, reverse DNS for every class, and an experimental key directory for Google-Agent. Nothing for auto browse in the user's Chrome.

IP lists

-   [developers.google.com/static/search/apis/ipranges/googlebot.json](https://developers.google.com/static/search/apis/ipranges/googlebot.json)
-   [developers.google.com/static/search/apis/ipranges/special-crawlers.json](https://developers.google.com/static/search/apis/ipranges/special-crawlers.json)
-   [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json)
-   [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json)
-   [developers.google.com/static/search/apis/ipranges/user-triggered-agents.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-agents.json)

Reverse DNS

crawl-\*.googlebot.com, geo-crawl-\*.geo.googlebot.com, rate-limited-proxy-\*.google.com, \*.gae.googleusercontent.com, google-proxy-\*.google.com

Key directory

https://agent.bot.goog/.well-known/http-message-signatures-directory (Google-Agent, subset of requests, experimental)

Cloudflare

not checked

Visa and Mastercard

Visa TAP: no public operator-level registration found (checked 2026-11-04). Mastercard Agent Pay: no public operator-level registration found (checked 2026-11-04).

## Block rates for Google tokens

| Token | Refused, top 10,000 | Named | Via * | All crawled |
| --- | --- | --- | --- | --- |
| Google-Extended | 16.0% | 12.6% | 3.4% | 12.1% |
| Google-CloudVertexBot | 10.9% | 7.1% | 3.8% | 8.3% |
| Googlebot | 2.8% | 0.5% | 2.3% | 2.1% |

Based on 5,938 top-10,000 sites with a parsed robots.txt; crawl of 2026-11-04.

## In the news

-   2026-09-15[Cloudflare adds Disallow AI Training and the Accountable label](https://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-lab/)
-   2026-08-19[Google retires the Google-NotebookLM user agent](https://agentreadycrawlers.com/news/2026-08-19-google-retires-the-google-notebooklm-user-agent/)

## Sources

-   [Verify requests from Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) read 2026-11-04
-   [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04

Last reviewed 2026-11-04