--- title: "Googlebot: user agent, token, block rate | Agent Ready Crawlers" description: "Googlebot from Google: exact user-agent string, robots.txt token Googlebot, search index purpose and how to verify it. 2.8% of top sites block it." canonical: https://agentreadycrawlers.com/crawlers/googlebot site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Googlebot: user agent, token, block rate | Agent Ready Crawlers > Googlebot from Google: exact user-agent string, robots.txt token Googlebot, search index purpose and how to verify it. 2.8% of top sites block it. Source: https://agentreadycrawlers.com/crawlers/googlebot Site: Agent Ready Crawlers Last reviewed: 2026-11-04 Search index crawler from Google # Googlebot In short Googlebot is Google's search index crawler. The token to write in robots.txt is Googlebot. Verify it by reverse DNS, then a forward lookup. 2.8 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Reverse DNS: crawl-\*\*\*-\*\*\*-\*\*\*-\*\*\*.googlebot.com or geo-crawl-\*\*\*.geo.googlebot.com, then forward-confirm. Googlebot: common crawlers always obey robots.txt rules when crawling automatically. Operator [Google](https://agentreadycrawlers.com/crawlers/operator/google/) Vendor profile: [Google](https://agentreadycrawlers.com/vendors/google/) Purpose Search index [All search index rows](https://agentreadycrawlers.com/crawlers/purpose/search-index/). Builds a search index that sends visitors back. Refusing it removes pages from that product. User-agent string `Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)` robots.txt token `Googlebot` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things. Respects robots.txt Yes, per operator Google: common crawlers always obey robots.txt rules when crawling automatically. [Source](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04. Verification Reverse DNS Reverse-resolve the source address, then forward-resolve the name and require the same address. [Documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests), read 2026-11-04. IP ranges [developers.google.com/static/search/apis/ipranges/googlebot.json](https://developers.google.com/static/search/apis/ipranges/googlebot.json) Rate behaviour Crawl rate can be reduced via Google's documented process; Google does not support Crawl-delay. Block rate **2.8%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `Googlebot` at the root (166 sites). 0.5% name it; 2.3% refuse it only through a \* group. Rank 31 of 31 tokens measured. Across all 8,624 parsed robots.txt files in the crawl: 2.1% (0.4% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/). Documentation [developers.google.com](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers), read 2026-11-04. Note. Reverse DNS: crawl-\*\*\*-\*\*\*-\*\*\*-\*\*\*.googlebot.com or geo-crawl-\*\*\*.geo.googlebot.com, then forward-confirm. Cloudflare classes Googlebot as an Accountable mixed-use crawler (2026-09-15); its "Block" setting now stops Googlebot entirely. ## Allow and refuse lines ### To allow Googlebot User-agent: Googlebot Allow: / ### To refuse Googlebot User-agent: Googlebot Disallow: / Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above. ## Google in the Verification Ledger Five IP lists, reverse DNS for every class, and an experimental key directory for Google-Agent. Nothing for auto browse in the user's Chrome. [All operators](https://agentreadycrawlers.com/verification/). IP lists - [developers.google.com/static/search/apis/ipranges/googlebot.json](https://developers.google.com/static/search/apis/ipranges/googlebot.json) - [developers.google.com/static/search/apis/ipranges/special-crawlers.json](https://developers.google.com/static/search/apis/ipranges/special-crawlers.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json) - [developers.google.com/static/search/apis/ipranges/user-triggered-agents.json](https://developers.google.com/static/search/apis/ipranges/user-triggered-agents.json) Reverse DNS crawl-\*.googlebot.com, geo-crawl-\*.geo.googlebot.com, rate-limited-proxy-\*.google.com, \*.gae.googleusercontent.com, google-proxy-\*.google.com Key directory https://agent.bot.goog/.well-known/http-message-signatures-directory (Google-Agent, subset of requests, experimental) Cloudflare not checked Visa TAP registry no public operator-level registration found (checked 2026-11-04) Mastercard Agent Pay no public operator-level registration found (checked 2026-11-04) ## In the news - 2026-09-15[Cloudflare adds Disallow AI Training and the Accountable label](https://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-lab/) ## Other Google rows - [Google-Extended (control token)](https://agentreadycrawlers.com/crawlers/google-extended/) Training token Google-Extended - [Google-CloudVertexBot](https://agentreadycrawlers.com/crawlers/google-cloudvertexbot/) Other token Google-CloudVertexBot - [GoogleOther](https://agentreadycrawlers.com/crawlers/googleother/) Other token GoogleOther - [Storebot-Google](https://agentreadycrawlers.com/crawlers/storebot-google/) Search index token Storebot-Google - [Google-Agent](https://agentreadycrawlers.com/crawlers/google-agent/) Task agent no token - [Gemini in Chrome (auto browse)](https://agentreadycrawlers.com/crawlers/gemini-in-chrome/) Task agent no token - [Google-GeminiNotebook (was Google-NotebookLM)](https://agentreadycrawlers.com/crawlers/google-gemininotebook/) Live answer fetch no token ## Sources - [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) read 2026-11-04 - [Verify requests from Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) read 2026-11-04 - [developers.google.com: googlebot (JSON)](https://developers.google.com/static/search/apis/ipranges/googlebot.json) read 2026-11-04 Last reviewed 2026-11-04