Bot from Agent Ready Crawlers

agent-readiness-crawl (the crawl behind this site)

In short

agent-readiness-crawl (the crawl behind this site) is Agent Ready Crawlers' bot, filed under Other. The token to write in robots.txt is agent-readiness-crawl. The operator publishes nothing a site can check. 4.2 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Run on 2026-11-04 from a single machine.

Operator

Agent Ready Crawlers

Purpose

Other All other rows. Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes.

User-agent string

Mozilla/5.0 (compatible; agent-readiness-crawl/1.0)

robots.txt token

agent-readiness-crawl Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.

Respects robots.txt

Partly, because one pass skipped the check

The main pass read /robots.txt first and fetched each of about 18 paths per domain only where the group for agent-readiness-crawl, or the * group, allowed it. A follow-up pass on 2026-11-04 fetched ten more well-known paths on 10,508 reachable domains with the same user agent but without a per-path robots.txt check; the crawl now checks every path for future runs. Source, read 2026-11-04.

Verification

Nothing published

The operator publishes no list, reverse DNS pattern or key directory a site owner can check. Documentation, read 2026-11-04.

IP ranges

Not published.

Rate behaviour

Main pass: about 18 plain GET requests per domain (robots.txt, the home page twice with one asking for text/markdown, one made-up missing path, llms.txt, llms-full.txt and 12 discovery paths), up to 100 domains at once, a 10 second timeout, no JavaScript, one request at a time per domain. Follow-up pass: ten more well-known paths on 10,508 reachable domains.

Block rate

4.2% of 5,938 top-10,000 sites with a parsed robots.txt disallow agent-readiness-crawl at the root (247 sites). 0.0% name it; 4.1% refuse it only through a * group. Rank 0 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 3.0% (0.0% named). Crawl of 2026-11-04; report.

Documentation

agentreadycrawlers.com, read 2026-11-04.

Note. Run on 2026-11-04 from a single machine. The user-agent string carries no contact URL and the crawl publishes no IP list, so it fails its own verification test: a site owner cannot confirm a request is ours except by this documentation. The follow-up pass skipped the per-path robots.txt check that the main pass applied, which is now fixed in the crawl for future runs. It stores only extracted fields and the first 200 words of a home page.

Allow and refuse lines

User-agent: agent-readiness-crawl
Allow: /

User-agent: agent-readiness-crawl
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

Last reviewed 2026-11-04