--- title: "The names sites write in robots.txt | Agent Ready Crawlers" description: "The user-agent names sites write most often in robots.txt, from 5,938 top-10,000 files, and which have no row in the Crawler Index. Crawl of 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/named-in-robots-txt site: "Agent Ready Crawlers" last_updated: 2026-11-05 date: 2026-11-04 section: "Crawlers" --- # The names sites write in robots.txt | Agent Ready Crawlers > The user-agent names sites write most often in robots.txt, from 5,938 top-10,000 files, and which have no row in the Crawler Index. Crawl of 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/named-in-robots-txt Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # The names sites write in robots.txt In short Sites write 3,034 distinct user-agent names in the 5,938 top-10,000 robots.txt files the crawl parsed. gptbot is first at 20.7 percent, and 13 of the 40 most written names have no row in the Crawler Index. No file among them names OraBot. A robots.txt group begins with one or more `User-agent` lines. Counting those names across the crawl shows which agents site owners think about, as opposed to which ones operators document. A name counts once per site, case-folded, and the `*` group is left out. The population is the 5,938 top-10,000 sites with a parsed robots.txt, out of 8,624 parsed in all 13,071 crawled domains. ## The fifteen most written names Share of 5,938 top-10,000 sites that write each name, crawl of 2026-11-04. The table below carries the first 40 names with the all-crawled counts. | Rank | Name | Sites, top 10,000 | Share | Sites, all crawled | In the Index | | --- | --- | --- | --- | --- | --- | | 1 | gptbot | 1,229 | 20.7% | 1,822 | GPTBot | | 2 | claudebot | 1,097 | 18.5% | 1,652 | ClaudeBot | | 3 | ccbot | 1,082 | 18.2% | 1,486 | CCBot | | 4 | google-extended | 1,029 | 17.3% | 1,561 | Google-Extended | | 5 | bytespider | 995 | 16.8% | 1,305 | Bytespider | | 6 | amazonbot | 830 | 14.0% | 1,092 | Amazonbot | | 7 | meta-externalagent | 820 | 13.8% | 1,091 | Meta-ExternalAgent | | 8 | applebot-extended | 815 | 13.7% | 1,196 | Applebot-Extended | | 9 | anthropic-ai | 802 | 13.5% | 1,111 | anthropic-ai legacy row | | 10 | perplexitybot | 757 | 12.7% | 1,269 | PerplexityBot | | 11 | cohere-ai | 745 | 12.5% | 977 | cohere-ai | | 12 | diffbot | 715 | 12.0% | 835 | Diffbot | | 13 | chatgpt-user | 714 | 12.0% | 1,185 | ChatGPT-User | | 14 | claude-web | 665 | 11.2% | 887 | No row | | 15 | omgili | 640 | 10.8% | 728 | omgili | | 16 | petalbot | 608 | 10.2% | 713 | PetalBot | | 17 | oai-searchbot | 590 | 9.9% | 995 | OAI-SearchBot | | 18 | googlebot | 536 | 9.0% | 769 | Googlebot | | 19 | baiduspider | 531 | 8.9% | 630 | No row | | 20 | google-cloudvertexbot | 443 | 7.5% | 514 | Google-CloudVertexBot | | 21 | facebookbot | 440 | 7.4% | 526 | FacebookBot legacy row | | 22 | yandex | 433 | 7.3% | 449 | No row | | 23 | bingbot | 428 | 7.2% | 670 | Bingbot | | 24 | adsbot-google | 424 | 7.1% | 512 | No row | | 25 | claude-user | 412 | 6.9% | 633 | Claude-User | | 26 | twitterbot | 406 | 6.8% | 490 | No row | | 27 | googleother | 398 | 6.7% | 513 | GoogleOther | | 28 | claude-searchbot | 396 | 6.7% | 611 | Claude-SearchBot | | 29 | ahrefsbot | 394 | 6.6% | 463 | No row | | 30 | magpie-crawler | 391 | 6.6% | 454 | No row | | 31 | icc-crawler | 380 | 6.4% | 437 | No row | | 32 | perplexity-user | 379 | 6.4% | 610 | Perplexity-User | | 33 | facebookexternalhit | 378 | 6.4% | 465 | FacebookExternalHit | | 34 | omgilibot | 370 | 6.2% | 403 | No row | | 35 | awariosmartbot | 369 | 6.2% | 428 | No row | | 36 | youbot | 357 | 6.0% | 456 | YouBot | | 37 | awariorssbot | 356 | 6.0% | 415 | No row | | 38 | cotoyogi | 341 | 5.7% | 398 | No row | | 39 | mediapartners-google | 323 | 5.4% | 328 | No row | | 40 | duckassistbot | 319 | 5.4% | 415 | DuckAssistBot | [Download the top 60 names for both populations as CSV](https://agentreadycrawlers.com/data/named-agents-2026-11-04.csv). ## Names with no row in the Index 13 of the 40 names have no row here: claude-web, baiduspider, yandex, adsbot-google, twitterbot, ahrefsbot, magpie-crawler, icc-crawler, omgilibot, awariosmartbot, awariorssbot, cotoyogi and mediapartners-google. The Index covers AI crawlers and agents, so search engines and SEO tools belong outside it. For the rest the gap is ours to close: a name that this many sites write is one more owners are asking about, and a row needs an operator page to be read first, so nothing is added without one. ## OraBot Of the 8,624 parsed files, none write the name `OraBot`, the user agent of [Ora's scanner](https://agentreadycrawlers.com/crawlers/orabot/). A site in the crawl can refuse it only through a \* group. That is expected for a bot whose page, read 2026-11-05, describes scans on demand and names no robots.txt route. ## Reading the counts A written name is a decision by a site owner or a copied list, and the crawl cannot tell which. A name can be written to allow as well as to refuse; the [Block Rate Report](https://agentreadycrawlers.com/block-rates/) has the refusals. The long tail is long: 3,034 distinct names in the top-10,000 files, and the 40 above are the part of it where more than a few percent of sites agree. ## Sources - [Block-rate method (this site)](https://agentreadycrawlers.com/method/#block-rate-method) read 2026-11-04 - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04 - [The Crawler Index (this site)](https://agentreadycrawlers.com/crawlers/) read 2026-11-04 Last reviewed 2026-11-05