---
title: "Google: crawler operator profile | Agent Ready Crawlers"
description: "Google: crawler operator profile covering Googlebot, Google-Extended, Google-Agent and 3 more. Standards, dated prices and limits, read 2026-11-05."
canonical: https://agentreadycrawlers.com/vendors/google
site: "Agent Ready Crawlers"
last_updated: 2026-11-05
section: "Vendors"
---

# Google: crawler operator profile | Agent Ready Crawlers

> Google: crawler operator profile covering Googlebot, Google-Extended, Google-Agent and 3 more. Standards, dated prices and limits, read 2026-11-05.

Source: https://agentreadycrawlers.com/vendors/google
Site: Agent Ready Crawlers
Last reviewed: 2026-11-05

Crawler operator

# Google

In short

Google separates its fetchers into three classes, each with a reverse DNS mask and a JSON range file, five files in all. A request never carries Google-Extended, which exists purely as a control word in robots.txt for Gemini training and grounding. Google-Agent traffic is partly signed with Web Bot Auth, an experiment described on 13 October 2026.

Type

[Crawler operator](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/)

Homepage

[google.com](https://www.google.com)

In one line

Google runs common crawlers such as Googlebot, special-case crawlers and user-triggered fetchers, verifiable by reverse DNS and five IP files, and is testing Web Bot Auth signatures on part of Google-Agent's traffic.

Last read

2026-11-05

## What Google does for this layer

### From the vendor's own pages

Google says its common crawlers, Googlebot among them, always obey robots.txt rules when crawling automatically. ([developers.google.com](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)) Google-Extended has no HTTP user agent of its own and works only as a robots.txt control token for Gemini model training and grounding; Google says it does not affect inclusion or ranking in Google Search. ([developers.google.com](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)) Google states it intends to support the AIPREF shared training preference signal, written "user-agent: \_\_usage\_ai-training", in early 2027. ([developers.google.com](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), dated 2026-10-30) Google's verification page sorts its requesters into common crawlers, special-case crawlers and user-triggered fetchers, each with its own reverse DNS mask and IP file. ([developers.google.com](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests))

Google says special-case crawlers such as AdsBot may or may not respect robots.txt, and that user-triggered fetchers ignore robots.txt because a user asked for the fetch. ([developers.google.com](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)) Google calls its Web Bot Auth use experimental, signs only a subset of Google-Agent requests under the identity https://agent.bot.goog, and tells sites to keep IP and reverse DNS checks as a fallback. ([developers.google.com](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth), dated 2026-10-13) Signed Google-Agent requests carry a Signature-Agent header set to g="https://agent.bot.goog", with the Signature and Signature-Input headers labelled g. ([developers.google.com](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth)) Google-Agent is the user agent for agents hosted on Google infrastructure that browse and act on user request, and it uses the ranges in user-triggered-agents.json. ([developers.google.com](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers))

Google's Gemini Notebook fetcher now sends the token Google-GeminiNotebook; the former Google-NotebookLM agent was supported until August 2026. ([developers.google.com](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers))

### What our crawl saw

Google's common-crawlers.json listed 317 prefixes, 170 IPv4 and 147 IPv6, with a creationTime of 2026-11-04 when fetched on 2026-11-05. ([developers.google.com](https://developers.google.com/static/crawling/ipranges/common-crawlers.json), dated 2026-11-04) Google's user-triggered-agents.json, the range file for Google-Agent, listed 20 prefixes (12 IPv4, 8 IPv6) on 2026-11-05. ([developers.google.com](https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json), dated 2026-11-04) agent.bot.goog served a key directory with five Ed25519 keys and the content type application/http-message-signatures-directory+json on 2026-11-05. ([agent.bot.goog](https://agent.bot.goog/.well-known/http-message-signatures-directory))

## Products and plans

| Product | What it does | Price model | From | As of |
| --- | --- | --- | --- | --- |
| Googlebot | Common crawler for Google Search, Images, Video, News and Discover. | No price recorded |
| Google-Extended | Robots.txt token, with no user agent string, that controls Gemini training and grounding use. | No price recorded |
| Google-Agent | User agent for Google-hosted agents acting on user request; some requests signed with Web Bot Auth. | No price recorded |
| Google-CloudVertexBot | Crawler for site-owner-requested crawls that build Vertex AI Agents. | No price recorded |
| Storebot-Google | Crawler for Google Shopping surfaces. | No price recorded |
| Google-GeminiNotebook | User-triggered fetcher for sources added to Gemini Notebook projects. | No price recorded |

## Standards

| Standard | Status | Evidence |
| --- | --- | --- |
| RFC 9309 (robots.txt) | Partial | Common crawlers always obey robots.txt for automatic crawls; special-case crawlers may not; user-triggered fetchers ignore it. (developers.google.com) |
| Web Bot Auth | Partial | Experimental; a subset of Google-Agent requests signed as https://agent.bot.goog. (developers.google.com) |
| Content Signals | No statement | Google names the AIPREF signal for early 2027; Content-Signals is not mentioned (read 2026-11-05). |
| Published IP list | Supports | Five published range files, including user-triggered-agents.json for Google-Agent. (developers.google.com) |
| Reverse DNS check | Supports | Masks under googlebot.com, google.com and googleusercontent.com with a forward-confirm step. (developers.google.com) |
| Cloudflare verified bots | No statement | Google's crawler pages do not mention Cloudflare's list (read 2026-11-05). |

## Our checks

| Check | Result | Method | Date |
| --- | --- | --- | --- |
| Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | agent.bot.goog: 200, application/http-message-signatures-directory+json, 5 keys | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 |
| common-crawlers.json | 317 prefixes (170 IPv4, 147 IPv6), creationTime 2026-11-04 | GET and count prefixes | 2026-11-05 |
| special-crawlers.json | 272 prefixes (136 IPv4, 136 IPv6), creationTime 2026-11-04 | GET and count prefixes | 2026-11-05 |
| user-triggered-agents.json | 20 prefixes (12 IPv4, 8 IPv6), creationTime 2026-11-04 | GET and count prefixes | 2026-11-05 |
| user-triggered-fetchers-google.json | 504 prefixes (252 IPv4, 252 IPv6), creationTime 2026-11-04 | GET and count prefixes | 2026-11-05 |

## What Google does not do

-   Google does not sign every Google-Agent request, so a missing signature does not prove a request is fake.
-   Google-Extended cannot be blocked at the server by user agent, because Google sends no Google-Extended string in requests.
-   Google's common crawler and user-triggered fetcher pages list no token for Gemini auto browse in Chrome (read 2026-11-05).

## Reading notes

Google's robots.txt disallows our crawl token for www.google.com/.well-known/, so that path was not probed for a key directory; the Web Bot Auth directory on agent.bot.goog was read instead.

## Related pages on this site

-   [All crawler operators](https://agentreadycrawlers.com/vendors/by-type/crawler-operator/) 31 profiles
-   [The full vendor list](https://agentreadycrawlers.com/vendors/) 73 profiles in four types
-   [Google in the Crawler Index](https://agentreadycrawlers.com/crawlers/operator/google/) 8 rows
-   [Googlebot](https://agentreadycrawlers.com/crawlers/googlebot/) token Googlebot
-   [Google-Extended (control token)](https://agentreadycrawlers.com/crawlers/google-extended/) token Google-Extended
-   [Google-CloudVertexBot](https://agentreadycrawlers.com/crawlers/google-cloudvertexbot/) token Google-CloudVertexBot
-   [GoogleOther](https://agentreadycrawlers.com/crawlers/googleother/) token GoogleOther
-   [Storebot-Google](https://agentreadycrawlers.com/crawlers/storebot-google/) token Storebot-Google
-   [Google-Agent](https://agentreadycrawlers.com/crawlers/google-agent/) no token
-   [Gemini in Chrome (auto browse)](https://agentreadycrawlers.com/crawlers/gemini-in-chrome/) no token
-   [Google-GeminiNotebook (was Google-NotebookLM)](https://agentreadycrawlers.com/crawlers/google-gemininotebook/) no token

## Sources

-   [google.com](https://www.google.com) read 2026-11-05
-   [Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) read 2026-11-05
-   [Verify Requests from Google Crawlers and Fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) read 2026-11-05
-   [Authenticate requests with Web Bot Auth (experimental)](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth) read 2026-11-05
-   [Google User-Triggered Fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) read 2026-11-05
-   [Google common-crawlers.json](https://developers.google.com/static/crawling/ipranges/common-crawlers.json) read 2026-11-05
-   [Google user-triggered-agents.json](https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json) read 2026-11-05
-   [agent.bot.goog Web Bot Auth key directory path (GET probe)](https://agent.bot.goog/.well-known/http-message-signatures-directory) read 2026-11-05
-   [Google special-crawlers.json](https://developers.google.com/static/crawling/ipranges/special-crawlers.json) read 2026-11-05
-   [Google user-triggered-fetchers-google.json](https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.json) read 2026-11-05

Last reviewed 2026-11-05