---
title: "agent-readiness-crawl: user agent and token | Agent Ready Crawlers"
description: "agent-readiness-crawl from Agent Ready Crawlers: exact user-agent string, robots.txt token agent-readiness-crawl, other purpose and how to verify it."
canonical: https://agentreadycrawlers.com/crawlers/agent-readiness-crawl
site: "Agent Ready Crawlers"
last_updated: 2026-11-04
section: "Crawlers"
---

# agent-readiness-crawl: user agent and token | Agent Ready Crawlers

> agent-readiness-crawl from Agent Ready Crawlers: exact user-agent string, robots.txt token agent-readiness-crawl, other purpose and how to verify it.

Source: https://agentreadycrawlers.com/crawlers/agent-readiness-crawl
Site: Agent Ready Crawlers
Last reviewed: 2026-11-04

Bot from Agent Ready Crawlers

# agent-readiness-crawl (the crawl behind this site)

In short

agent-readiness-crawl (the crawl behind this site) is Agent Ready Crawlers' bot, filed under Other. The token to write in robots.txt is agent-readiness-crawl. The operator publishes nothing a site can check. 4.2 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Run on 2026-11-04 from a single machine.

Operator

Agent Ready Crawlers

Purpose

Other [All other rows](https://agentreadycrawlers.com/crawlers/purpose/other/). Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes.

User-agent string

`Mozilla/5.0 (compatible; agent-readiness-crawl/1.0)`

robots.txt token

`agent-readiness-crawl` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.

Respects robots.txt

Partly, because one pass skipped the check

The main pass read /robots.txt first and fetched each of about 18 paths per domain only where the group for agent-readiness-crawl, or the \* group, allowed it. A follow-up pass on 2026-11-04 fetched ten more well-known paths on 10,508 reachable domains with the same user agent but without a per-path robots.txt check; the crawl now checks every path for future runs. [Source](https://agentreadycrawlers.com/method/#own-crawl), read 2026-11-04.

Verification

Nothing published

The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://agentreadycrawlers.com/method/#own-crawl), read 2026-11-04.

IP ranges

Not published.

Rate behaviour

Main pass: about 18 plain GET requests per domain (robots.txt, the home page twice with one asking for text/markdown, one made-up missing path, llms.txt, llms-full.txt and 12 discovery paths), up to 100 domains at once, a 10 second timeout, no JavaScript, one request at a time per domain. Follow-up pass: ten more well-known paths on 10,508 reachable domains.

Block rate

**4.2%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `agent-readiness-crawl` at the root (247 sites). 0.0% name it; 4.1% refuse it only through a \* group. Rank 0 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 3.0% (0.0% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/).

Documentation

[agentreadycrawlers.com](https://agentreadycrawlers.com/method/#own-crawl), read 2026-11-04.

Note. Run on 2026-11-04 from a single machine. The user-agent string carries no contact URL and the crawl publishes no IP list, so it fails its own verification test: a site owner cannot confirm a request is ours except by this documentation. The follow-up pass skipped the per-path robots.txt check that the main pass applied, which is now fixed in the crawl for future runs. It stores only extracted fields and the first 200 words of a home page.

## Allow and refuse lines

### To allow agent-readiness-crawl

User-agent: agent-readiness-crawl
Allow: /

### To refuse agent-readiness-crawl

User-agent: agent-readiness-crawl
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

## Sources

-   [agentreadycrawlers.com](https://agentreadycrawlers.com/method/#own-crawl) read 2026-11-04

Last reviewed 2026-11-04