Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Browsing agent from Google

Google-Agent

In short

Google-Agent is Google's browsing agent. Google-Agent has no robots.txt token, so robots.txt cannot address it and the crawl cannot measure a block rate. It signs requests with Web Bot Auth, so a signature can be checked. Google lists it as a user-triggered fetcher; these fetchers generally ignore robots.txt because the fetch was requested by a user.

Operator
Google Vendor profile: Google
Purpose
Task agent All task agent rows. A browsing agent that clicks, fills forms and finishes a job for a person.
User-agent string
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36
robots.txt token
None. No robots.txt line can address this agent.
Respects robots.txt

May not apply (user-triggered)

Google lists it as a user-triggered fetcher; these fetchers generally ignore robots.txt because the fetch was requested by a user. Source, read 2026-11-04.

Verification

Web Bot Auth signature

Requests carry an HTTP message signature that can be checked against the operator's published key directory. Documentation, read 2026-11-04.

IP ranges
developers.google.com/static/search/apis/ipranges/user-triggered-agents.json
Rate behaviour
User-triggered; no rate published.
Block rate

None: the agent has no token, so a robots.txt block rate does not exist.

Documentation
developers.google.com, read 2026-11-04.

Note. Used by agents hosted on Google infrastructure to navigate and act on user request. A subset of requests is signed with Web Bot Auth as https://agent.bot.goog (experimental); Google says keep IP and reverse-DNS checks as not every request is signed. user-triggered-agents.json creationTime 2026-03-03.

Allow and refuse lines

This agent has no robots.txt token, so there are no lines to copy. Verify its signature in your WAF or edge rules instead, then allow or refuse the verified identity.

Refusing or allowing in robots.txt states a preference. For this row the operator or its documentation says robots.txt may not govern the fetch, so enforcement belongs in your WAF.

Google in the Verification Ledger

Five IP lists, reverse DNS for every class, and an experimental key directory for Google-Agent. Nothing for auto browse in the user's Chrome. All operators.

IP lists
Reverse DNS
crawl-*.googlebot.com, geo-crawl-*.geo.googlebot.com, rate-limited-proxy-*.google.com, *.gae.googleusercontent.com, google-proxy-*.google.com
Key directory
https://agent.bot.goog/.well-known/http-message-signatures-directory (Google-Agent, subset of requests, experimental)
Cloudflare
not checked
Visa TAP registry
no public operator-level registration found (checked 2026-11-04)
Mastercard Agent Pay
no public operator-level registration found (checked 2026-11-04)

In the news

Other Google rows

Sources

Last reviewed 2026-11-04