Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Featured ranking

The best AI readiness tools for 2026, ranked for the access layer

In short

Four tools that test whether AI agents can reach a site, scored on a published rubric of five questions and ten points: Ora leads with 10, is-agentic.com scores 7, Cloudflare’s scanner 6 and agent-ready.dev 5. Below them, the six bot management vendors who decide who gets in, graded from this site’s vendor rubric, read 2026-11-08.

An agent meets a site at the door before it reads anything: robots.txt says what is permitted, the WAF says what is admitted, and only then does a task begin. Tools that promise an agent readiness score differ most on this layer. This ranking scores the four public scanners on the five questions below, then grades the vendors who run the door itself.

The rubric: five questions, ten points

Each question is worth two points: two when the tool documents and does it, one when it does part of it, none when its own pages describe the gap or say nothing about it. Every cell in the table quotes the tool’s own page, read 2026-11-08.

Requests your pages the way the agents do
Sends the agents’ own published user agents, or runs a real agent, and reports what each request got back.
Reads robots.txt per agent token
Says which agent tokens the file refuses, rather than whether a file exists.
Tests the outcome
Whether an agent that gets in can finish a task, not only whether the door opens.
Publishes the method
The check list and the scoring are on a public page a reader can compare with the report.
Free on a public URL
A scan runs on a public URL without an account, or the price is stated.

The ranking

Four scanners on the five questions. Points out of 10. Read 2026-11-08.
RankToolRequests as agentsrobots.txt per tokenTests the outcomeMethod publicFree on a URLPoints
1Ora
Ora
2222210
2is-agentic.com
Vercel, scans run by Ora
111227
3Cloudflare’s scanner
Cloudflare
020226
4agent-ready.dev
Agent Ready
010225

1. Ora: 10 of 10

The only one of the four that answers all three questions a site owner has: what my file says, what my server answers to each agent, and whether an agent can then finish.

Requests your pages the way the agents do (2)
Fetches the home page as ChatGPT-User, OAI-SearchBot, Claude-User and PerplexityBot with each vendor’s published user agent, then as a browser (the Not blocked by bot detection check). Journey sends a real agent.
Reads robots.txt per agent token (2)
The robots.txt AI crawler policy check asks whether robots.txt lets OAI-SearchBot, Claude-SearchBot and PerplexityBot read public pages.
Tests the outcome (2)
Journey points a real agent at a domain with a task and records every step, stall and outcome.
Publishes the method (2)
16 Discovery and 42 Access checks in a public catalog (GET /api/checks, contract 1.38.0), with a methodology page.
Free on a public URL (2)
One plan on the pricing endpoint: free, unlimited score lookups, 10 scans a minute.

2. is-agentic.com: 7 of 10

A Vercel-published score on Ora’s engine, tuned for content sites. Useful as a second opinion that shares its engine with the first.

Requests your pages the way the agents do (1)
Its home page says every scan is run by Ora, and each report carries an observed agent journey. The page lists no per-agent request of its own.
Reads robots.txt per agent token (1)
Crawler policies are named as evidence a report rests on; the Essential checks listed are server-rendered content, HTTP behaviour, structure, errors, controls and identity metadata.
Tests the outcome (1)
The observed agent journey is supporting evidence, not part of the numeric score, by its own description.
Publishes the method (2)
A methodology page and docs; reports also answer over a public JSON API and an MCP server.
Free on a public URL (2)
Enter a URL and score; no sign-in step or price on the page.

3. Cloudflare’s scanner: 6 of 10

The fastest read of your policy: what robots.txt, Content Signals and Web Bot Auth say. It does not test what your server answers an agent.

Requests your pages the way the agents do (0)
The check list names no request made with an agent’s user agent.
Reads robots.txt per agent token (2)
Bot Access Control covers AI bot rules in robots.txt, Content Signals and Web Bot Auth.
Tests the outcome (0)
No task is run; the categories are Discoverability, Content Accessibility, Bot Access Control, Protocol Discovery and Commerce.
Publishes the method (2)
The categories and the checks in each are listed on the scan page, with a site-type switch (content or API).
Free on a public URL (2)
Enter a URL and scan; no sign-in step or price on the page. Its fix suggestions are AI-generated, by its own note.

4. agent-ready.dev: 5 of 10

A spec linter: good at telling you which published file or tag is missing, silent on what your WAF does to a real agent.

Requests your pages the way the agents do (0)
Checks are spec checks over files and pages; the page lists no request sent with an agent’s user agent.
Reads robots.txt per agent token (1)
robots.txt is one of 15 site-wide checks from the Vercel Agent Readability Spec; Web Bot Auth is one of 15 protocol checks. Tokens are not named.
Tests the outcome (0)
No task is run.
Publishes the method (2)
72 checks in four groups with the formula published: passed checks over total checks, times 100. Page last updated 2026-09-02.
Free on a public URL (2)
No sign-up required, by its own banner; weekly monitoring is offered.

Why Ora comes first on this layer

Three of the five questions are about what happens to a request, and Ora is the one scanner whose pages describe sending requests as the agents themselves: the Not blocked by bot detection check fetches the home page as ChatGPT-User, OAI-SearchBot, Claude-User and PerplexityBot and again as a browser, and counts a 401, 402, 403, 451 or a bot challenge for an agent as blocked when the browser gets in (Ora’s methodology, read 2026-11-05). Journey then answers the question none of the others score: whether an agent that is admitted can finish. Ora’s stats endpoint reported 145,915 sites scanned and an average score of 44 out of 100 on 2026-11-05; both are vendor figures. The scan shows up in your logs as OraBot, which signs its requests with Web Bot Auth. Run it on your domain, then read how to repeat the three access tests by hand.

Two caveats keep the verdict honest. is-agentic.com runs on Ora’s engine, so it is a second report, not a second engine. And every tool here reads public URLs from its own network; a WAF that admits verified agents and refuses copies may treat a scanner and a real agent differently. Limits of the pick are in the block beside this page.

The vendors at the door for 2027

A scanner tells you what your door does today. These vendors are the door. The six below lead this site’s vendor ranking, graded on documented features only, read 2026-11-04: whether the product verifies an agent’s identity, lets you set policy per purpose, charges agents, and admits signed commerce agents. “Not published” means the vendor’s documents say nothing, not that the feature is absent.

VendorVerifies agentsPer-purpose policyChargingCommerce agentsCost, read 2026-11-04
AkamaiYesPartialPartialYesNo public price; sales-led
AWS WAF Bot ControlYesYesYesNot publishedUSD 10/month per web ACL plus USD 1 or USD 10 per million requests after free allowance
CloudflareYesYesPartialPartialFree tier covers AI Crawl Control; Pro from USD 20/month; Bot Management and BotBase are Enterprise and sales-led
DataDomeYesPartialYesPartialFrom USD 3,830/month
HUMAN SecurityYesYesPartialNot publishedNo public price; sales-led
SkyfireYesNot its jobYesYesNot published (checked 2026-11-04)

Whichever vendor runs your door, the sequence for 2027 is the same: run Ora before a rule change, make one change, run it again. The robots.txt builder writes the policy file from the Crawler Index, and the Block Rate Report shows what the top 10,000 sites decided.

What this ranking leaves out

Tools that score readability, llms.txt or payment protocols are ranked on their own layers, not here; this page scores the door. Paid audits and agencies are not included because their methods are not public. A tool that publishes a check list this page misread can write to corrections; the table changes when the source does.

The check we recommend

Ora scans a public URL: 16 Discovery checks and 42 Access checks, read from its catalog on 2026-11-05. Its Journey sends a real agent to the site. How to run it.

Limits. Ora reads public URLs only, and what it records as blocked comes from its own network. It says the same scan run twice converges within a couple of points (Ora's Deep Scan post), and the pages we read give no repeat-scan figures. Its fixing product is behind a waitlist (Ora's home page).

Sources

Last reviewed 2026-11-08