Training crawler from Google

Google-Extended (control token)

In short

Google-Extended (control token) is Google's training crawler. The token to write in robots.txt is Google-Extended. The operator publishes nothing a site can check. 16.0 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Google says it intends to support the AIPREF training preference signal (user-agent: __usage_ai-training) in early 2027.

Operator

Google Vendor profile: Google

Purpose

Training All training rows. Collects content to train or tune a model. Refusing it is the training opt-out.

User-agent string

No exact string is published. What the operator or a third party says: None: crawling uses existing Google user agent strings; the token is a control only

robots.txt token

Google-Extended Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.

Respects robots.txt

Yes, per operator

Google: Google-Extended manages whether crawled content may be used for training future Gemini models and for grounding in Gemini Apps and Vertex AI; it does not affect Search inclusion or ranking. Source, read 2026-11-04.

Verification

Nothing published

The operator publishes no list, reverse DNS pattern or key directory a site owner can check. Documentation, read 2026-11-04.

IP ranges

Not published.

Rate behaviour

No separate crawl.

Block rate

16.0% of 5,938 top-10,000 sites with a parsed robots.txt disallow Google-Extended at the root (949 sites). 12.6% name it; 3.4% refuse it only through a * group. Rank 5 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 12.1% (9.6% named). Crawl of 2026-11-04; report.

Documentation

developers.google.com, read 2026-11-04.

Note. Google says it intends to support the AIPREF training preference signal (user-agent: __usage_ai-training) in early 2027.

Allow and refuse lines

User-agent: Google-Extended
Allow: /

User-agent: Google-Extended
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

Google in the Verification Ledger

Five IP lists, reverse DNS for every class, and an experimental key directory for Google-Agent. Nothing for auto browse in the user's Chrome. All operators.

IP lists

• developers.google.com/static/search/apis/ipranges/googlebot.json
• developers.google.com/static/search/apis/ipranges/special-crawlers.json
• developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json
• developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json
• developers.google.com/static/search/apis/ipranges/user-triggered-agents.json

Reverse DNS

crawl-*.googlebot.com, geo-crawl-*.geo.googlebot.com, rate-limited-proxy-*.google.com, *.gae.googleusercontent.com, google-proxy-*.google.com

Key directory

https://agent.bot.goog/.well-known/http-message-signatures-directory (Google-Agent, subset of requests, experimental)

Cloudflare

not checked

Visa TAP registry

no public operator-level registration found (checked 2026-11-04)

Mastercard Agent Pay

no public operator-level registration found (checked 2026-11-04)

Other Google rows

• Googlebot Search index token Googlebot
• Google-CloudVertexBot Other token Google-CloudVertexBot
• GoogleOther Other token GoogleOther
• Storebot-Google Search index token Storebot-Google
• Google-Agent Task agent no token
• Gemini in Chrome (auto browse) Task agent no token
• Google-GeminiNotebook (was Google-NotebookLM) Live answer fetch no token

Last reviewed 2026-11-04