https://agentreadycrawlers.com/ The Crawler Index: 70 AI crawlers and agents with exact tokens, verification and block rates from 13,071 crawled sites, plus the vendors at the door. Wed, 04 Nov 2026 08:00:00 GMT https://agentreadycrawlers.com/guides/403-to-ai-agentshttps://agentreadycrawlers.com/guides/403-to-ai-agentsThu, 05 Nov 2026 08:00:00 GMTWhat a 403 to an AI agent does, per Ora's lab, how many top home pages answer 401 or 403 in our crawl, and how to find your own refusals. Read 2026-11-05. https://agentreadycrawlers.com/guides/ora-checks-for-bot-accesshttps://agentreadycrawlers.com/guides/ora-checks-for-bot-accessThu, 05 Nov 2026 08:00:00 GMTThe seven Ora checks that read robots.txt, user agents and the home page, with Ora's fix advice and the matching crawl numbers. Catalog read 2026-11-05. https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-sitehttps://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-siteThu, 05 Nov 2026 08:00:00 GMTHow to test whether AI agents can reach your site: robots.txt, home page status per agent user agent, logs and the Ora scan, with curl commands. https://agentreadycrawlers.com/vendorshttps://agentreadycrawlers.com/vendorsThu, 05 Nov 2026 08:00:00 GMTThe full vendor list: 73 crawler operators, bot management, verification and licensing vendors with standards, dated prices and sources. Read 2026-11-05. https://agentreadycrawlers.com/block-rateshttps://agentreadycrawlers.com/block-ratesWed, 04 Nov 2026 08:00:00 GMTThe Block Rate Report: the share of top-10,000 sites that disallow each AI crawler token and the share of home pages answering 401 or 403. 13,071 rows. https://agentreadycrawlers.com/block-rates/2026-11-04https://agentreadycrawlers.com/block-rates/2026-11-04Wed, 04 Nov 2026 08:00:00 GMTBlock Rate Report edition 2026-11-04: three charts and tables for token, category and the 401 and 403 share, from a crawl of 13,071 domains. https://agentreadycrawlers.com/block-rates/edge-and-bot-headershttps://agentreadycrawlers.com/block-rates/edge-and-bot-headersWed, 04 Nov 2026 08:00:00 GMTThe 401 and 403 share of home pages split by the CDN, host and bot-product headers they carried, from a crawl of 13,071 domains. Crawl of 2026-11-04. https://agentreadycrawlers.com/block-rates/training-and-searchhttps://agentreadycrawlers.com/block-rates/training-and-searchWed, 04 Nov 2026 08:00:00 GMTHow many top sites refuse training crawlers but keep AI search open, how many refuse both, and how many refuse neither. Crawl of 2026-11-04. https://agentreadycrawlers.com/crawlers/agentic-browsershttps://agentreadycrawlers.com/crawlers/agentic-browsersWed, 04 Nov 2026 08:00:00 GMTEvery task-agent and shopping-agent row in the Crawler Index: token, user-agent string, verification and what the operator says about robots.txt. https://agentreadycrawlers.com/crawlers/named-in-robots-txthttps://agentreadycrawlers.com/crawlers/named-in-robots-txtWed, 04 Nov 2026 08:00:00 GMTThe user-agent names sites write most often in robots.txt, from 5,938 top-10,000 files, and which have no row in the Crawler Index. Crawl of 2026-11-04. https://agentreadycrawlers.com/datahttps://agentreadycrawlers.com/dataWed, 04 Nov 2026 08:00:00 GMTDownload the Crawler Index, the Block Rate Report, the Verification Ledger and two crawl slices as CSV and JSON. 13,071 crawl rows. https://agentreadycrawlers.com/guides/allow-search-refuse-traininghttps://agentreadycrawlers.com/guides/allow-search-refuse-trainingWed, 04 Nov 2026 08:00:00 GMTHow to allow search and answer bots while refusing training: the tokens to name, the control tokens that never crawl, and the Bing gap. https://agentreadycrawlers.com/guides/cloudflare-verified-botshttps://agentreadycrawlers.com/guides/cloudflare-verified-botsWed, 04 Nov 2026 08:00:00 GMTHow Cloudflare's verified bot list works: BotBase submission, Web Bot Auth or IP validation, behaviour classes and the rules you can write on them. https://agentreadycrawlers.com/guides/crawler-versus-browsing-agenthttps://agentreadycrawlers.com/guides/crawler-versus-browsing-agentWed, 04 Nov 2026 08:00:00 GMTThe difference between a crawler and a browsing agent: schedule against task, token against none, and what that means for robots.txt. https://agentreadycrawlers.com/guides/let-tap-and-agent-pay-throughhttps://agentreadycrawlers.com/guides/let-tap-and-agent-pay-throughWed, 04 Nov 2026 08:00:00 GMTHow to let Visa TAP and Mastercard Agent Pay agents through: check whether your CDN verifies Web Bot Auth before you write custom rules. https://agentreadycrawlers.com/guides/read-agent-traffic-in-logshttps://agentreadycrawlers.com/guides/read-agent-traffic-in-logsWed, 04 Nov 2026 08:00:00 GMTHow to read AI crawler traffic in your server logs: match the token, check the source address against the operator's list, and log Signature-Agent. https://agentreadycrawlers.com/guides/test-what-a-403-doeshttps://agentreadycrawlers.com/guides/test-what-a-403-doesWed, 04 Nov 2026 08:00:00 GMTHow to test what a 403 does to an AI answer about you: fetch your pages as each crawler, compare status codes, then ask the assistant. https://agentreadycrawlers.com/guides/tokens-versus-user-agent-stringshttps://agentreadycrawlers.com/guides/tokens-versus-user-agent-stringsWed, 04 Nov 2026 08:00:00 GMTHow robots.txt tokens differ from user-agent strings: what you write in robots.txt, what arrives in the request, and tokens that never appear in logs. https://agentreadycrawlers.com/guides/verify-web-bot-authhttps://agentreadycrawlers.com/guides/verify-web-bot-authWed, 04 Nov 2026 08:00:00 GMTHow to verify Web Bot Auth signatures: read Signature-Agent, fetch the key directory, check the RFC 9421 signature and keep IP checks as a fallback. https://agentreadycrawlers.com/guides/what-content-signals-changehttps://agentreadycrawlers.com/guides/what-content-signals-changeWed, 04 Nov 2026 08:00:00 GMTWhat Content-Signals changes: the search, ai-input and ai-train lines in robots.txt, Cloudflare's defaults and why they enforce nothing. https://agentreadycrawlers.com/guides/what-pay-per-crawl-payshttps://agentreadycrawlers.com/guides/what-pay-per-crawl-paysWed, 04 Nov 2026 08:00:00 GMTWhat pay per crawl pays and who is paying: Cloudflare, AWS WAF and TollBit compared, and why no payout totals are public yet. https://agentreadycrawlers.com/guides/which-bots-to-allow-by-site-typehttps://agentreadycrawlers.com/guides/which-bots-to-allow-by-site-typeWed, 04 Nov 2026 08:00:00 GMTWhich AI bots to allow for a publisher, a docs site, a store, a logged-in app or a closed site, with a robots.txt generated from the Crawler Index. https://agentreadycrawlers.com/verificationhttps://agentreadycrawlers.com/verificationWed, 04 Nov 2026 08:00:00 GMTThe Verification Ledger: which of 31 operators publish IP lists, reverse DNS patterns or Web Bot Auth key directories, with URL and date read. https://agentreadycrawlers.com/verification/programmes-comparedhttps://agentreadycrawlers.com/verification/programmes-comparedWed, 04 Nov 2026 08:00:00 GMTIP lists, reverse DNS, Web Bot Auth, Cloudflare signed agents, Visa TAP and Mastercard Agent Pay compared: what a site checks and which Index rows use each. https://agentreadycrawlers.com/newshttps://agentreadycrawlers.com/newsMon, 02 Nov 2026 08:00:00 GMT15 dated news items on AI crawlers, agents and bot management, from 2026-08-19 to 2026-11-02, each tied to the Crawler Index rows it changes. Read 2026-11-04. https://agentreadycrawlers.com/news/2026-11-02-signed-agent-metadata-header-proposed-for-web-bot-authhttps://agentreadycrawlers.com/news/2026-11-02-signed-agent-metadata-header-proposed-for-web-bot-authMon, 02 Nov 2026 08:00:00 GMTSigned-Agent-Metadata header proposed for Web Bot Auth. A new draft would carry signed metadata about an agent beside its signature. Dated 2026-11-02. https://agentreadycrawlers.com/news/2026-10-29-vercel-splits-ai-crawlers-from-ai-assistantshttps://agentreadycrawlers.com/news/2026-10-29-vercel-splits-ai-crawlers-from-ai-assistantsThu, 29 Oct 2026 08:00:00 GMTVercel splits AI crawlers from AI assistants. Vercel verifies known bots by IP, reverse DNS and Web Bot Auth. Dated 2026-10-29. https://agentreadycrawlers.com/news/2026-10-29-web-bot-auth-working-group-draft-reaches-01https://agentreadycrawlers.com/news/2026-10-29-web-bot-auth-working-group-draft-reaches-01Thu, 29 Oct 2026 08:00:00 GMTWeb Bot Auth working-group draft reaches -01. A registry and Signature Agent card draft (-04) followed on 2026-10-27. Dated 2026-10-29. https://agentreadycrawlers.com/news/2026-10-27-openai-updates-the-chatgpt-user-ip-listhttps://agentreadycrawlers.com/news/2026-10-27-openai-updates-the-chatgpt-user-ip-listTue, 27 Oct 2026 08:00:00 GMTOpenAI updates the ChatGPT-User IP list. openai.com/chatgpt-user.json carried creationTime 2026-10-27 when read on 2026-11-04. https://agentreadycrawlers.com/news/2026-10-22-ietf-drafts-propose-a-common-ip-range-format-and-crawler-beshttps://agentreadycrawlers.com/news/2026-10-22-ietf-drafts-propose-a-common-ip-range-format-and-crawler-besThu, 22 Oct 2026 08:00:00 GMTIETF drafts propose a common IP-range format and crawler best practices: one for publishing IP ranges, one for crawler conduct. Dated 2026-10-22. https://agentreadycrawlers.com/news/2026-10-21-anthropic-s-claude-duet-starts-browsing-with-a-claude-agenthttps://agentreadycrawlers.com/news/2026-10-21-anthropic-s-claude-duet-starts-browsing-with-a-claude-agentWed, 21 Oct 2026 08:00:00 GMTAnthropic's Claude Duet starts browsing with a Claude-Agent token. Claude Duet launched as a waitlist research preview for US Pro and Max subscribers. https://agentreadycrawlers.com/news/2026-10-20-anthropic-refreshes-its-crawler-ip-listhttps://agentreadycrawlers.com/news/2026-10-20-anthropic-refreshes-its-crawler-ip-listTue, 20 Oct 2026 08:00:00 GMTAnthropic refreshes its crawler IP list. Anthropic warns that IP blocking may stop its bots reading robots.txt. Dated 2026-10-20. https://agentreadycrawlers.com/news/2026-10-15-datadome-q3-report-25-billion-ai-agent-requestshttps://agentreadycrawlers.com/news/2026-10-15-datadome-q3-report-25-billion-ai-agent-requestsThu, 15 Oct 2026 08:00:00 GMTDataDome Q3 report: 25 billion AI agent requests. It counted 29.1 million Muse requests in under two weeks without self-identification. https://agentreadycrawlers.com/news/2026-10-01-datadome-joins-experian-agent-trusthttps://agentreadycrawlers.com/news/2026-10-01-datadome-joins-experian-agent-trustThu, 01 Oct 2026 08:00:00 GMTDataDome joins Experian Agent Trust. DataDome already lists Web Bot Auth, KYA tokens, IP lists and reverse DNS as identification methods. https://agentreadycrawlers.com/news/2026-09-17-arcjet-launches-agent-runtime-securityhttps://agentreadycrawlers.com/news/2026-09-17-arcjet-launches-agent-runtime-securityThu, 17 Sep 2026 08:00:00 GMTArcjet launches Agent Runtime Security. Its inbound bot protection still verifies known bots by IP and reverse DNS. Dated 2026-09-17. https://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-labhttps://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-labTue, 15 Sep 2026 08:00:00 GMTCloudflare adds Disallow AI Training and the Accountable label: a no-training line in robots.txt that keeps mixed-use crawlers in search. Dated 2026-09-15. https://agentreadycrawlers.com/news/2026-09-08-meta-launches-muse-a-personal-agent-that-does-not-identify-ihttps://agentreadycrawlers.com/news/2026-09-08-meta-launches-muse-a-personal-agent-that-does-not-identify-iTue, 08 Sep 2026 08:00:00 GMTMeta launches Muse, a personal agent that does not identify itself. Meta released Muse in the US on iOS, Android, the web and WhatsApp. https://agentreadycrawlers.com/news/2026-09-02-human-publishes-agent-trust-levelshttps://agentreadycrawlers.com/news/2026-09-02-human-publishes-agent-trust-levelsWed, 02 Sep 2026 08:00:00 GMTHUMAN publishes Agent Trust Levels. Customers set permissions per action such as account creation and checkout. Dated 2026-09-02. https://agentreadycrawlers.com/news/2026-08-28-cloudflare-opens-botbase-for-operatorshttps://agentreadycrawlers.com/news/2026-08-28-cloudflare-opens-botbase-for-operatorsFri, 28 Aug 2026 08:00:00 GMTCloudflare opens BotBase for Operators. Cloudflare says accurate entries decide whether a bot earns and keeps Verified status. https://agentreadycrawlers.com/news/2026-08-19-google-retires-the-google-notebooklm-user-agenthttps://agentreadycrawlers.com/news/2026-08-19-google-retires-the-google-notebooklm-user-agentWed, 19 Aug 2026 08:00:00 GMTGoogle retires the Google-NotebookLM user agent. The page, last updated 2026-08-19, also lists Google-Agent and its experimental Web Bot Auth identity.