Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Glossary

Google-Extended

In short

Google's control token for whether crawled content may train Gemini models and ground Gemini Apps and Vertex AI answers. It does not change Search inclusion or ranking. 16.0 percent of top-10,000 sites refuse it, 12.6 percent by naming it. It has no user-agent string, so it never appears in logs. Example: A Disallow for Google-Extended leaves Googlebot crawling for Search.

Definition

Google's control token for whether crawled content may train Gemini models and ground Gemini Apps and Vertex AI answers. It does not change Search inclusion or ranking.

Sourced example

A Disallow for Google-Extended leaves Googlebot crawling for Search. Source, read 2026-11-04.

How it shows up here

16.0 percent of top-10,000 sites refuse it, 12.6 percent by naming it. It has no user-agent string, so it never appears in logs.

Previous: Disallow AI Training. Next: HTTP 402 Payment Required. All 41 terms.

Sources

Last reviewed 2026-11-04