Search index crawler from Google

Storebot-Google

In short

Storebot-Google is Google's search index crawler. The token to write in robots.txt is Storebot-Google. Verify it by reverse DNS, then a forward lookup. The crawl of 2026-11-04 did not measure Storebot-Google. Relevant to commerce sites: blocking it removes Shopping surfaces. Its user-agent entry: Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36. Storebot-Google obeys robots.txt, per Google.

Operator

Google Vendor profile: Google

Purpose

Search index All search index rows. Builds a search index that sends visitors back. Refusing it removes pages from that product.

User-agent string

Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36

robots.txt token

Storebot-Google Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.

Respects robots.txt

Yes, per operator

Common crawler: obeys robots.txt, per Google. Preferences affect all surfaces of Google Shopping. Source, read 2026-11-04.

Verification

Reverse DNS

Reverse-resolve the source address, then forward-resolve the name and require the same address. Documentation, read 2026-11-04.

IP ranges

developers.google.com/static/search/apis/ipranges/googlebot.json

Rate behaviour

Not published separately.

Block rate

Not measured: the crawl checked 31 AI and search tokens, and Storebot-Google is not one of them.

Documentation

developers.google.com, read 2026-11-04.

Note. Relevant to commerce sites: blocking it removes Shopping surfaces.

Allow and refuse lines

User-agent: Storebot-Google
Allow: /

User-agent: Storebot-Google
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

Google in the Verification Ledger

Five IP lists, reverse DNS for every class, and an experimental key directory for Google-Agent. Nothing for auto browse in the user's Chrome. All operators.

IP lists

• developers.google.com/static/search/apis/ipranges/googlebot.json
• developers.google.com/static/search/apis/ipranges/special-crawlers.json
• developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json
• developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json
• developers.google.com/static/search/apis/ipranges/user-triggered-agents.json

Reverse DNS

crawl-*.googlebot.com, geo-crawl-*.geo.googlebot.com, rate-limited-proxy-*.google.com, *.gae.googleusercontent.com, google-proxy-*.google.com

Key directory

https://agent.bot.goog/.well-known/http-message-signatures-directory (Google-Agent, subset of requests, experimental)

Cloudflare

not checked

Visa TAP registry

no public operator-level registration found (checked 2026-11-04)

Mastercard Agent Pay

no public operator-level registration found (checked 2026-11-04)

Other Google rows

• Googlebot Search index token Googlebot
• Google-Extended (control token) Training token Google-Extended
• Google-CloudVertexBot Other token Google-CloudVertexBot
• GoogleOther Other token GoogleOther
• Google-Agent Task agent no token
• Gemini in Chrome (auto browse) Task agent no token
• Google-GeminiNotebook (was Google-NotebookLM) Live answer fetch no token

Last reviewed 2026-11-04