The names sites write in robots.txt

In short

Sites write 3,034 distinct user-agent names in the 5,938 top-10,000 robots.txt files the crawl parsed. gptbot is first at 20.7 percent, and 13 of the 40 most written names have no row in the Crawler Index. No file among them names OraBot.

A robots.txt group begins with one or more User-agent lines. Counting those names across the crawl shows which agents site owners think about, as opposed to which ones operators document. A name counts once per site, case-folded, and the * group is left out. The population is the 5,938 top-10,000 sites with a parsed robots.txt, out of 8,624 parsed in all 13,071 crawled domains.

The fifteen most written names

Download the top 60 names for both populations as CSV.

Names with no row in the Index

13 of the 40 names have no row here: claude-web, baiduspider, yandex, adsbot-google, twitterbot, ahrefsbot, magpie-crawler, icc-crawler, omgilibot, awariosmartbot, awariorssbot, cotoyogi and mediapartners-google. The Index covers AI crawlers and agents, so search engines and SEO tools belong outside it. For the rest the gap is ours to close: a name that this many sites write is one more owners are asking about, and a row needs an operator page to be read first, so nothing is added without one.

OraBot

Of the 8,624 parsed files, none write the name OraBot, the user agent of Ora's scanner. A site in the crawl can refuse it only through a * group. That is expected for a bot whose page, read 2026-11-05, describes scans on demand and names no robots.txt route.

Reading the counts

A written name is a decision by a site owner or a copied list, and the crawl cannot tell which. A name can be written to allow as well as to refuse; the Block Rate Report has the refusals. The long tail is long: 3,034 distinct names in the top-10,000 files, and the 40 above are the part of it where more than a few percent of sites agree.

Last reviewed 2026-11-05