# Agent Ready Crawlers > The Crawler Index: 70 AI crawlers and agents with exact tokens, verification and block rates from 13,071 crawled sites, plus the vendors at the door. Agent Ready Crawlers is a reference on the AI crawlers and agents that visit websites and the vendors that decide whether they get in. Start with the Crawler Index at /crawlers/ (one row and one profile page per crawler or agent: exact user-agent string, robots.txt token, purpose, the operator's statement on robots.txt, verification method, allow and refuse lines, block rate). The Block Rate Report at /block-rates/ gives the share of sites that disallow each token, from a crawl of 13,071 domains on 2026-11-04. The Verification Ledger at /verification/ lists which operators publish IP lists, reverse DNS patterns and key directories. Datasets are at /data/ as CSV and JSON. Every fact carries a source URL and the date it was read. The same rows are queryable through the MCP server at /mcp, the REST API described at /openapi.json and the NLWeb endpoint at /ask, all read-only and documented at /developers. Every page has a markdown twin at the same path plus /index.md (or ask for it with Accept: text/markdown). The whole site in one file: https://agentreadycrawlers.com/llms-full.txt. Sitemap: https://agentreadycrawlers.com/sitemap.xml. Feed: https://agentreadycrawlers.com/feed.xml. ## Home - [The Crawler Index](https://agentreadycrawlers.com/): The Crawler Index: 70 AI crawlers and agents with exact tokens, verification and block rates from 13,071 crawled sites, plus the vendors at the door. ## Publication - [About Agent Ready Crawlers | Agent Ready Crawlers](https://agentreadycrawlers.com/about): About Agent Ready Crawlers: what the publication covers, who it is written for, how the Crawler Index, the Block Rate Report and the vendor ranking are made. - [Contact | Agent Ready Crawlers](https://agentreadycrawlers.com/contact): How to reach the editors of Agent Ready Crawlers: send corrections, a missing crawler or agent, or a source for a vendor grade to the editor mailbox. - [Corrections and known gaps | Agent Ready Crawlers](https://agentreadycrawlers.com/corrections): Corrections log for Agent Ready Crawlers, with the known gaps in the 2026-11-04 edition: what we could not confirm and where each gap is noted in the Index. - [Editorial policy | Agent Ready Crawlers](https://agentreadycrawlers.com/editorial-policy): Editorial policy: how sources are chosen and dated, how vendor claims and Ora's findings are labelled, how grades change and how corrections are logged. - [Privacy | Agent Ready Crawlers](https://agentreadycrawlers.com/privacy): Privacy on Agent Ready Crawlers: the site sets no cookies, runs no third-party scripts and keeps what you type in the robots.txt builder in your browser. - [Terms | Agent Ready Crawlers](https://agentreadycrawlers.com/terms): Terms of use for Agent Ready Crawlers: how to cite the pages and datasets, what the data covers and how it is dated, and how to reach the editors about changes. ## Block rates - [Block Rate Report: who refuses which crawler | Agent Ready Crawlers](https://agentreadycrawlers.com/block-rates): The Block Rate Report: the share of top-10,000 sites that disallow each AI crawler token and the share of home pages answering 401 or 403. 13,071 rows. - [Block Rate Report, edition 2026-11-04 | Agent Ready Crawlers](https://agentreadycrawlers.com/block-rates/2026-11-04): Block Rate Report edition 2026-11-04: three charts and tables for token, category and the 401 and 403 share, from a crawl of 13,071 domains. - [401 and 403 by edge and bot-product header | Agent Ready Crawlers](https://agentreadycrawlers.com/block-rates/edge-and-bot-headers): The 401 and 403 share of home pages split by the CDN, host and bot-product headers they carried, from a crawl of 13,071 domains. Crawl of 2026-11-04. - [Training and search policy in robots.txt | Agent Ready Crawlers](https://agentreadycrawlers.com/block-rates/training-and-search): How many top sites refuse training crawlers but keep AI search open, how many refuse both, and how many refuse neither. Crawl of 2026-11-04. ## Comparisons Section index: https://agentreadycrawlers.com/compare/llms.txt - [Akamai vs Cloudflare | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/akamai-vs-cloudflare): Akamai vs Cloudflare: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [Akamai vs HUMAN Security | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/akamai-vs-human-security): Akamai vs HUMAN Security: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [Arcjet vs Cloudflare | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/arcjet-vs-cloudflare): Arcjet vs Cloudflare: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [Arcjet vs Vercel | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/arcjet-vs-vercel-botid): Arcjet vs Vercel: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [Auth0 for AI Agents vs Skyfire | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/auth0-ai-agents-vs-skyfire): Auth0 for AI Agents vs Skyfire: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [AWS WAF Bot Control vs Akamai | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/aws-waf-bot-control-vs-akamai): AWS WAF Bot Control vs Akamai: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [AWS WAF Bot Control vs DataDome | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/aws-waf-bot-control-vs-datadome): AWS WAF Bot Control vs DataDome: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [Cloudflare vs AWS WAF Bot Control | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/cloudflare-vs-aws-waf-bot-control): Cloudflare vs AWS WAF Bot Control: rubric grades side by side, the sourced difference and verdicts by reader type. Read 2026-11-04. - [Cloudflare vs DataDome | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/cloudflare-vs-datadome): Cloudflare vs DataDome: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [Cloudflare vs HUMAN Security | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/cloudflare-vs-human-security): Cloudflare vs HUMAN Security: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [Cloudflare vs Netlify | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/cloudflare-vs-netlify): Cloudflare vs Netlify: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [Cloudflare vs Vercel | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/cloudflare-vs-vercel-botid): Cloudflare vs Vercel: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [DataDome vs Akamai | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/datadome-vs-akamai): DataDome vs Akamai: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [DataDome vs HUMAN Security | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/datadome-vs-human-security): DataDome vs HUMAN Security: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [DataDome vs Kasada | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/datadome-vs-kasada): DataDome vs Kasada: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [Fastly vs Akamai | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/fastly-vs-akamai): Fastly vs Akamai: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [Fastly vs Cloudflare | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/fastly-vs-cloudflare): Fastly vs Cloudflare: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [HUMAN Security vs Kasada | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/human-security-vs-kasada): HUMAN Security vs Kasada: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [Imperva vs Cloudflare | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/imperva-vs-cloudflare): Imperva vs Cloudflare: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [Imperva vs DataDome | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/imperva-vs-datadome): Imperva vs DataDome: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [Kasada vs Akamai | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/kasada-vs-akamai): Kasada vs Akamai: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [Kasada vs Vercel | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/kasada-vs-vercel-botid): Kasada vs Vercel: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [Skyfire vs Cloudflare | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/skyfire-vs-cloudflare): Skyfire vs Cloudflare: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [Skyfire vs Stytch IsAgent | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/skyfire-vs-stytch-isagent): Skyfire vs Stytch IsAgent: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [Skyfire vs TollBit | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/skyfire-vs-tollbit): Skyfire vs TollBit: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [Stytch IsAgent vs Auth0 for AI Agents | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/stytch-isagent-vs-auth0-ai-agents): Stytch IsAgent vs Auth0 for AI Agents: rubric grades side by side, the sourced difference and verdicts by reader type. Read 2026-11-04. - [TollBit vs AWS WAF Bot Control | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/tollbit-vs-aws-waf-bot-control): TollBit vs AWS WAF Bot Control: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [TollBit vs Cloudflare | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/tollbit-vs-cloudflare): TollBit vs Cloudflare: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [Vercel vs AWS WAF Bot Control | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/vercel-botid-vs-aws-waf-bot-control): Vercel vs AWS WAF Bot Control: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. - [Vercel vs Netlify | Agent Ready Crawlers](https://agentreadycrawlers.com/compare/vercel-botid-vs-netlify): Vercel vs Netlify: rubric grades side by side, the sourced difference and verdicts for platform engineers, security engineers and publisher ops. ## Crawlers Section index: https://agentreadycrawlers.com/crawlers/llms.txt - [Crawler Index: every row | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers): All 70 AI crawlers and agents in the Crawler Index with operator, token, purpose, verification and block rate, and a profile page per row. Read 2026-11-05. - [agent-readiness-crawl: user agent and token | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/agent-readiness-crawl): agent-readiness-crawl from Agent Ready Crawlers: exact user-agent string, robots.txt token agent-readiness-crawl, other purpose and how to verify it. - [Amazon Bedrock AgentCore Browser profile | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/agentcore-browser): Amazon Bedrock AgentCore Browser from Amazon Web Services: no robots.txt token, purpose and what a site can check. Read 2026-11-04. - [Agentic browsers and how they identify | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/agentic-browsers): Every task-agent and shopping-agent row in the Crawler Index: token, user-agent string, verification and what the operator says about robots.txt. - [Buy for Me: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/amazon-buy-for-me): Buy for Me from Amazon: no robots.txt token, shopping agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. - [Rufus: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/amazon-rufus): Rufus from Amazon: no robots.txt token, shopping agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. - [Amazonbot: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/amazonbot): Amazonbot from Amazon: exact user-agent string, robots.txt token Amazonbot, training purpose and how to verify it. 15.6% of top sites block it. - [Amzn-SearchBot: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/amzn-searchbot): Amzn-SearchBot from Amazon: exact user-agent string, robots.txt token Amzn-SearchBot, search index purpose and how to verify it. Read 2026-11-04. - [Amzn-User: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/amzn-user): Amzn-User from Amazon: exact user-agent string, robots.txt token Amzn-User, live answer fetch purpose and how to verify it. Read 2026-11-04. - [Anchor Browser: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/anchor-browser): Anchor Browser from Anchor: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. - [anthropic-ai: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/anthropic-ai): anthropic-ai from Anthropic: robots.txt token anthropic-ai, training purpose and what a site can check; no exact user-agent string is published. - [Apify Website Content Crawler profile | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/apify): Apify Website Content Crawler from Apify: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. - [Applebot: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/applebot): Applebot from Apple: exact user-agent string, robots.txt token Applebot, search index purpose and how to verify it. 14.8% of top sites block it. - [Applebot-Extended: user agent and token | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/applebot-extended): Applebot-Extended from Apple: robots.txt token Applebot-Extended, training purpose and what a site can check; no exact user-agent string is published. - [Bingbot: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/bingbot): Bingbot from Microsoft: exact user-agent string, robots.txt token bingbot, search index purpose and how to verify it. 3.1% of top sites block it. - [Browser Use: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/browser-use): Browser Use from Browser Use: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. - [Browserbase cloud browsers profile | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/browserbase): Browserbase cloud browsers from Browserbase: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. - [Bytespider: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/bytespider): Bytespider from ByteDance: robots.txt token Bytespider, other purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. - [CCBot: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/ccbot): CCBot from Common Crawl: exact user-agent string, robots.txt token CCBot, training purpose and how to verify it. 19.4% of top sites block it. Read 2026-11-04. - [ChatGPT agent: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/chatgpt-agent): ChatGPT agent from OpenAI: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. - [ChatGPT Atlas: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/chatgpt-atlas): ChatGPT Atlas from OpenAI: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. - [ChatGPT-User: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/chatgpt-user): ChatGPT-User from OpenAI: exact user-agent string, robots.txt token ChatGPT-User, live answer fetch purpose and how to verify it. 9.7% of top sites block it. - [Claude computer use: user agent and token | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/claude-computer-use): Claude computer use from Anthropic: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. - [Claude Duet cloud browser profile | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/claude-duet): Claude Duet cloud browser from Anthropic: robots.txt token Claude-Agent, purpose and what a site can check. Read 2026-11-04. - [Claude in Chrome: user agent and token | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/claude-in-chrome): Claude in Chrome from Anthropic: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. - [Claude-SearchBot: user agent and token | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/claude-searchbot): Claude-SearchBot from Anthropic: robots.txt token Claude-SearchBot, search index purpose and what a site can check; no exact user-agent string is published. - [Claude-User: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/claude-user): Claude-User from Anthropic: robots.txt token Claude-User, live answer fetch purpose and what a site can check; no exact user-agent string is published. - [ClaudeBot: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/claudebot): ClaudeBot from Anthropic: robots.txt token ClaudeBot, training purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. - [Cloudflare Browser Rendering profile | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/cloudflare-browser-rendering): Cloudflare Browser Rendering from Cloudflare: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. - [cohere-ai: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/cohere-ai): cohere-ai from Cohere: robots.txt token cohere-ai, other purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. - [Browse with Copilot: user agent and token | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/copilot-actions): Browse with Copilot from Microsoft: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. - [Diffbot: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/diffbot): Diffbot from Diffbot: robots.txt token Diffbot, other purpose and what a site can check; no exact user-agent string is published. 15.7% of top sites block it. - [DuckAssistBot: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/duckassistbot): DuckAssistBot from DuckDuckGo: exact user-agent string, robots.txt token DuckAssistBot, live answer fetch purpose and how to verify it. Read 2026-11-04. - [DuckDuckBot: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/duckduckbot): DuckDuckBot from DuckDuckGo: exact user-agent string, robots.txt token DuckDuckBot, search index purpose and how to verify it. Read 2026-11-04. - [ExaSearchBot: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/exasearchbot): ExaSearchBot from Exa: robots.txt token ExaSearchBot, search index purpose and what a site can check; no exact user-agent string is published. - [FacebookBot: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/facebookbot): FacebookBot from Meta: robots.txt token FacebookBot, other purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. - [FacebookExternalHit: user agent and token | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/facebookexternalhit): FacebookExternalHit from Meta: exact user-agent string, robots.txt token facebookexternalhit, other purpose and how to verify it. Read 2026-11-04. - [FirecrawlAgent: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/firecrawl): FirecrawlAgent from Firecrawl: robots.txt token FirecrawlAgent, other purpose and what a site can check; no exact user-agent string is published. - [Gemini in Chrome: user agent and token | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/gemini-in-chrome): Gemini in Chrome from Google: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. - [Google-Agent: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/google-agent): Google-Agent from Google: exact user-agent string, no robots.txt token, task agent purpose and how to verify it. Read 2026-11-04. - [Google-CloudVertexBot: user agent and token | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/google-cloudvertexbot): Google-CloudVertexBot from Google: robots.txt token Google-CloudVertexBot, other purpose and what a site can check; no exact user-agent string is published. - [Google-Extended: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/google-extended): Google-Extended from Google: robots.txt token Google-Extended, training purpose and what a site can check; no exact user-agent string is published. - [Google-GeminiNotebook: user agent and token | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/google-gemininotebook): Google-GeminiNotebook from Google: exact user-agent string, no robots.txt token, live answer fetch purpose and how to verify it. Read 2026-11-04. - [Googlebot: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/googlebot): Googlebot from Google: exact user-agent string, robots.txt token Googlebot, search index purpose and how to verify it. 2.8% of top sites block it. - [GoogleOther: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/googleother): GoogleOther from Google: exact user-agent string, robots.txt token GoogleOther, other purpose and how to verify it. Read 2026-11-04. - [Goose: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/goose): Goose from Block: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. - [GPTBot: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/gptbot): GPTBot from OpenAI: exact user-agent string, robots.txt token GPTBot, training purpose and how to verify it. 18.5% of top sites block it. Read 2026-11-04. - [Grok Bot: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/grok-bot): Grok Bot from xAI: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. - [ImagesiftBot: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/imagesiftbot): ImagesiftBot from Hive (ImageSift): exact user-agent string, robots.txt token ImagesiftBot, other purpose and how to verify it. 9.5% of top sites block it. - [Manus cloud browser: user agent and token | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/manus): Manus cloud browser from Manus: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. - [Meta-ExternalAds: user agent and token | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/meta-externalads): Meta-ExternalAds from Meta: exact user-agent string, robots.txt token meta-externalads, other purpose and how to verify it. Read 2026-11-04. - [Meta-ExternalAgent: user agent and token | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/meta-externalagent): Meta-ExternalAgent from Meta: exact user-agent string, robots.txt token meta-externalagent, training purpose and how to verify it. Read 2026-11-04. - [Meta-ExternalFetcher: user agent and token | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/meta-externalfetcher): Meta-ExternalFetcher from Meta: exact user-agent string, robots.txt token meta-externalfetcher, live answer fetch purpose and how to verify it. - [Muse: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/meta-muse): Muse from Meta: no robots.txt token, shopping agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. - [Meta-WebIndexer: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/meta-webindexer): Meta-WebIndexer from Meta: exact user-agent string, robots.txt token meta-webindexer, search index purpose and how to verify it. Read 2026-11-04. - [MicrosoftPreview: user agent and token | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/microsoftpreview): MicrosoftPreview from Microsoft: exact user-agent string, robots.txt token MicrosoftPreview, other purpose and how to verify it. 4.0% of top sites block it. - [MistralAI-Index: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/mistralai-index): MistralAI-Index from Mistral AI: exact user-agent string, robots.txt token MistralAI-Index, search index purpose and how to verify it. Read 2026-11-04. - [MistralAI-Training: user agent and token | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/mistralai-training): MistralAI-Training from Mistral AI: exact user-agent string, robots.txt token MistralAI-Training, training purpose and how to verify it. Read 2026-11-04. - [MistralAI-User: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/mistralai-user): MistralAI-User from Mistral AI: exact user-agent string, robots.txt token MistralAI-User, live answer fetch purpose and how to verify it. Read 2026-11-04. - [The names sites write in robots.txt | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/named-in-robots-txt): The user-agent names sites write most often in robots.txt, from 5,938 top-10,000 files, and which have no row in the Crawler Index. Crawl of 2026-11-04. - [OAI-AdsBot: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/oai-adsbot): OAI-AdsBot from OpenAI: exact user-agent string, robots.txt token OAI-AdsBot, other purpose and how to verify it. Read 2026-11-04. - [OAI-SearchBot: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/oai-searchbot): OAI-SearchBot from OpenAI: exact user-agent string, robots.txt token OAI-SearchBot, search index purpose and how to verify it. 7.7% of top sites block it. - [omgili: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/omgili): omgili from Webz.io: exact user-agent string, robots.txt token omgili, other purpose and how to verify it. 15.7% of top sites block it. Read 2026-11-04. - [Amazon crawlers and agents | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/operator/amazon): Amazon in the Crawler Index: 5 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. - [Anthropic crawlers and agents | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/operator/anthropic): Anthropic in the Crawler Index: 7 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. - [Apple crawlers and agents | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/operator/apple): Apple in the Crawler Index: 2 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. IP list and reverse DNS. - [Browserbase crawlers and agents | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/operator/browserbase): Browserbase in the Crawler Index: 2 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. - [DuckDuckGo crawlers and agents | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/operator/duckduckgo): DuckDuckGo in the Crawler Index: 2 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. Two IP lists. - [Google crawlers and agents | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/operator/google): Google in the Crawler Index: 8 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. - [Meta crawlers and agents | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/operator/meta): Meta in the Crawler Index: 7 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. - [Microsoft crawlers and agents | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/operator/microsoft): Microsoft in the Crawler Index: 3 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. - [Mistral AI crawlers and agents | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/operator/mistral-ai): Mistral AI in the Crawler Index: 3 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. - [OpenAI crawlers and agents | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/operator/openai): OpenAI in the Crawler Index: 6 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. - [Perplexity crawlers and agents | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/operator/perplexity): Perplexity in the Crawler Index: 4 crawlers and agents with tokens, purposes, verification method and block rates, read 2026-11-04. - [OraBot: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/orabot): OraBot from Ora: exact user-agent string, robots.txt token OraBot, other purpose and how to verify it. Verification: Web Bot Auth. Read 2026-11-05. - [Comet: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/perplexity-comet): Comet from Perplexity: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. - [Perplexity Computer: user agent and token | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/perplexity-computer): Perplexity Computer from Perplexity: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. - [Perplexity-User: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/perplexity-user): Perplexity-User from Perplexity: exact user-agent string, robots.txt token Perplexity-User, live answer fetch purpose and how to verify it. Read 2026-11-04. - [PerplexityBot: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/perplexitybot): PerplexityBot from Perplexity: exact user-agent string, robots.txt token PerplexityBot, search index purpose and how to verify it. Read 2026-11-04. - [PetalBot: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/petalbot): PetalBot from Huawei: exact user-agent string, robots.txt token PetalBot, search index purpose and how to verify it. 14.1% of top sites block it. - [Live answer fetch crawlers and agents | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch): Live answer fetch in the Crawler Index: 8 AI crawlers and agents from 8 operators, with tokens, verification and block rates. Read 2026-11-04. - [Other crawlers and agents | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/purpose/other): Other in the Crawler Index: 15 AI crawlers and agents from 12 operators, with tokens, verification and block rates. Read 2026-11-04. - [Search index crawlers and agents | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/purpose/search-index): Search index in the Crawler Index: 15 AI crawlers and agents from 14 operators, with tokens, verification and block rates. Read 2026-11-04. - [Shopping agent crawlers and agents | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/purpose/shopping-agent): Shopping agent in the Crawler Index: 3 AI crawlers and agents from 2 operators, with tokens, verification and block rates. Read 2026-11-04. - [Task agent crawlers and agents | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/purpose/task-agent): Task agent in the Crawler Index: 20 AI crawlers and agents from 14 operators, with tokens, verification and block rates. Read 2026-11-04. - [Training crawlers and agents | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/purpose/training): Training in the Crawler Index: 9 AI crawlers and agents from 8 operators, with tokens, verification and block rates. Read 2026-11-04. - [Stagehand: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/stagehand): Stagehand from Browserbase: no robots.txt token, task agent purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. - [Storebot-Google: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/storebot-google): Storebot-Google from Google: exact user-agent string, robots.txt token Storebot-Google, search index purpose and how to verify it. Read 2026-11-04. - [Timpibot: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/timpibot): Timpibot from Timpi: robots.txt token Timpibot, search index purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. - [YouBot: user agent, token, block rate | Agent Ready Crawlers](https://agentreadycrawlers.com/crawlers/youbot): YouBot from You.com: robots.txt token YouBot, search index purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04. ## Data - [Data downloads: CSV and JSON | Agent Ready Crawlers](https://agentreadycrawlers.com/data): Download the Crawler Index, the Block Rate Report, the Verification Ledger and two crawl slices as CSV and JSON. 13,071 crawl rows. ## Developers - [API and MCP server | Agent Ready Crawlers](https://agentreadycrawlers.com/developers): How to query the crawler data: the MCP server at /mcp, the REST API described by openapi.json and the NLWeb ask endpoint, all read-only and open. ## FAQ - [FAQ: AI crawlers and agents | Agent Ready Crawlers](https://agentreadycrawlers.com/faq): 33 questions on AI crawlers and agents: ClaudeBot, GPTBot, tokens, verification, pay per crawl and bot management, each answered with a source. ## Glossary Section index: https://agentreadycrawlers.com/glossary/llms.txt - [Glossary of crawler and agent terms | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary): 41 glossary terms for AI crawler traffic: tokens, verification, Web Bot Auth, Content-Signals, pay per crawl and bot management, each with a sourced example. - [Accountable designation: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/accountable-designation): Accountable designation: Cloudflare's label for crawlers whose operators offer a training opt-out and say it does not affect search. Read 2026-11-04. - [Agentic browser: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/agentic-browser): Agentic browser: A consumer browser with a built-in agent that can act on pages, such as Comet, Gemini in Chrome or Browse with Copilot. - [ai-input: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/ai-input): ai-input: The Content Signals value for using content as live input to an AI answer, such as grounding or retrieval. Leaving it blank states no preference. - [AIPREF training signal: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/aipref-training-signal): AIPREF training signal: An IETF AI Preferences vocabulary for usage preferences. The robots.txt builder on this site does not write it. - [Applebot-Extended: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/applebot-extended): Applebot-Extended: Apple's control token for whether Applebot data trains Apple foundation models. It crawls nothing itself. - [Bot management: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/bot-management): Bot management: products that classify automated traffic and allow, challenge, rate-limit or block it, now with separate handling for AI crawlers and agents. - [Bot paywall: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/bot-paywall): Bot paywall: A redirect target where AI bots are told the price for content and must present a valid token to get it. Read 2026-11-04. - [Bot Preference Sync: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/bot-preference-sync): Bot Preference Sync: The Cloudflare feature that writes a zone's crawler preferences into the robots.txt it serves. Read 2026-11-04. - [Bot score: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/bot-score): Bot score: A numeric likelihood that a request is automated, used in rules. Signed traffic can be given a fixed score. Read 2026-11-04. - [BotBase: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/botbase): BotBase: Cloudflare's directory of known bots and agents, with an operator portal (launched 2026-08-28) for submissions, status and edits. - [Browsing agent: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/browsing-agent): Browsing agent: Software that drives a browser to complete a task for a person, clicking, typing and checking out. Read 2026-11-04. - [Content-Signal: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/content-signal): Content-Signal: A robots.txt line from the Content Signals Policy that states yes or no for three uses, search, ai-input and ai-train. - [Crawl-delay: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/crawl-delay): Crawl-delay: A non-standard robots.txt line asking a crawler to wait a number of seconds between requests. Read 2026-11-04. - [crawler-price header: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/crawler-price-header): crawler-price header: The response header Cloudflare pay per crawl sends with a 402 to state the per-request price. Read 2026-11-04. - [Disallow AI Training: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/disallow-ai-training): Disallow AI Training: a Cloudflare setting, launched 2026-09-15, that publishes a no-training preference in robots.txt and keeps search crawlers allowed. - [Google-Extended: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/google-extended): Google-Extended: Google's control token for whether crawled content may train Gemini models and ground Gemini Apps and Vertex AI answers. - [HTTP 402 Payment Required: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/http-402): HTTP 402 Payment Required: An HTTP status code reserved since the 1990s and now used by pay-per-crawl and x402 to tell a client that access needs payment. - [HTTP Message Signatures: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/http-message-signatures): HTTP Message Signatures: RFC 9421, a standard for signing selected components of an HTTP message, carried in Signature and Signature-Input headers. - [IP range JSON: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/ip-range-json): IP range JSON: A JSON file of CIDR prefixes an operator publishes for one crawler, usually with a creationTime field. Read 2026-11-04. - [Key directory: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/key-directory): Key directory: the JSON file at /.well-known/http-message-signatures-directory that lists a signer's public keys for Web Bot Auth verification. - [Know Your Agent (KYA): definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/know-your-agent): Know Your Agent (KYA): An identity scheme in which an issuer vouches for an agent, its platform and the user it represents, carried as a signed token. - [KYAPay: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/kyapay): KYAPay: The token form that combines KYA identity with payment intent and committed funds, so a seller can verify and charge in one step. - [Mastercard Agent Pay: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/mastercard-agent-pay): Mastercard Agent Pay: Mastercard's programme for registered agents to transact, which signs agent traffic with Web Bot Auth so merchants and CDNs. - [Mixed-use crawler: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/mixed-use-crawler): Mixed-use crawler: A single crawler whose content serves both search and AI training, so blocking it to refuse training also removes the site from search. - [noarchive: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/noarchive): noarchive: A robots meta directive. Bing reads it today as the opt-out from AI training; Amazon reads it as "do not use the page for model training". - [nosnippet: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/nosnippet): nosnippet: A robots meta directive that limits snippets. Apple applies it to opt content out of broad world knowledge answers. - [Pay per crawl: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/pay-per-crawl): Pay per crawl: Cloudflare's closed-beta feature that lets a zone charge AI crawlers per request using HTTP 402, with Cloudflare as Merchant of Record. - [Reverse DNS verification: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/reverse-dns-verification): Reverse DNS verification: Checking a client IP by looking up its hostname, confirming the operator's domain. Read 2026-11-04. - [RFC 9309: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/rfc-9309): RFC 9309: the IETF Robots Exclusion Protocol standard of September 2022, which defines groups, Allow and Disallow rules and longest-match precedence. - [robots.txt: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/robots-txt): robots.txt: A plain-text file at the root of a host that tells crawlers which paths they may fetch, grouped by user-agent token. - [Robots.txt token: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/robots-txt-token): Robots.txt token: The name a crawler answers to in a User-agent line. It can differ from the string the crawler sends in requests. - [Search-index crawler: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/search-index-crawler): Search-index crawler: A crawler that builds an index used to surface and cite pages in search or AI answers. Read 2026-11-04. - [Signature-Agent header: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/signature-agent-header): Signature-Agent header: The request header in Web Bot Auth that names the URL of the signer's key directory, telling the site where to fetch public keys. - [Signed agent: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/signed-agent): Signed agent: Cloudflare's class for agents directed by end users whose requests are signed with Web Bot Auth and who follow its signed agent policy. - [Spoofed crawler: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/spoofed-crawler): Spoofed crawler: A request that copies a known crawler's user agent to get treatment meant for the real bot. Read 2026-11-04. - [Training crawler: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/training-crawler): Training crawler: A crawler whose fetched content may be used to train AI models. 9 Index rows are filed under Training. - [User-agent string: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/user-agent-string): User-agent string: The User-Agent HTTP header a client sends. Anyone can copy it, so operators pair it with IP lists, reverse DNS or signatures. - [User-triggered fetcher: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/user-triggered-fetcher): User-triggered fetcher: A fetch made because a person asked an assistant something, such as reading a linked page. Read 2026-11-04. - [Verified bot: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/verified-bot): Verified bot: A bot that a security provider has confirmed belongs to its stated operator, by IP validation or signature. Read 2026-11-04. - [Visa Trusted Agent Protocol: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/visa-trusted-agent-protocol): Visa Trusted Agent Protocol: Visa's specification for agents to present a signature with timestamps that merchants verify to tell a credentialed. - [Web Bot Auth: definition | Agent Ready Crawlers](https://agentreadycrawlers.com/glossary/web-bot-auth): Web Bot Auth: An IETF protocol in which bots and agents sign HTTP requests with HTTP Message Signatures and publish their public keys. Read 2026-11-04. ## Guides Section index: https://agentreadycrawlers.com/guides/llms.txt - [Guides for crawler and agent traffic | Agent Ready Crawlers](https://agentreadycrawlers.com/guides): 14 guides on AI crawler traffic: logs, robots.txt tokens, Web Bot Auth, signed commerce agents, pay per crawl, Content-Signals and testing access. - [403 to AI agents: what the crawl and Ora show | Agent Ready Crawlers](https://agentreadycrawlers.com/guides/403-to-ai-agents): What a 403 to an AI agent does, per Ora's lab, how many top home pages answer 401 or 403 in our crawl, and how to find your own refusals. Read 2026-11-05. - [Allow search, refuse training | Agent Ready Crawlers](https://agentreadycrawlers.com/guides/allow-search-refuse-training): How to allow search and answer bots while refusing training: the tokens to name, the control tokens that never crawl, and the Bing gap. - [How Cloudflare's verified bot list works | Agent Ready Crawlers](https://agentreadycrawlers.com/guides/cloudflare-verified-bots): How Cloudflare's verified bot list works: BotBase submission, Web Bot Auth or IP validation, behaviour classes and the rules you can write on them. - [Crawler versus browsing agent | Agent Ready Crawlers](https://agentreadycrawlers.com/guides/crawler-versus-browsing-agent): The difference between a crawler and a browsing agent: schedule against task, token against none, and what that means for robots.txt. - [Letting TAP and Agent Pay agents through | Agent Ready Crawlers](https://agentreadycrawlers.com/guides/let-tap-and-agent-pay-through): How to let Visa TAP and Mastercard Agent Pay agents through: check whether your CDN verifies Web Bot Auth before you write custom rules. - [The Ora checks that touch bot access | Agent Ready Crawlers](https://agentreadycrawlers.com/guides/ora-checks-for-bot-access): The seven Ora checks that read robots.txt, user agents and the home page, with Ora's fix advice and the matching crawl numbers. Catalog read 2026-11-05. - [How to read agent traffic in your logs | Agent Ready Crawlers](https://agentreadycrawlers.com/guides/read-agent-traffic-in-logs): How to read AI crawler traffic in your server logs: match the token, check the source address against the operator's list, and log Signature-Agent. - [What a 403 does to an AI answer | Agent Ready Crawlers](https://agentreadycrawlers.com/guides/test-what-a-403-does): How to test what a 403 does to an AI answer about you: fetch your pages as each crawler, compare status codes, then ask the assistant. - [Test whether AI agents can reach your site | Agent Ready Crawlers](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site): How to test whether AI agents can reach your site: robots.txt, home page status per agent user agent, logs and the Ora scan, with curl commands. - [Tokens versus user-agent strings | Agent Ready Crawlers](https://agentreadycrawlers.com/guides/tokens-versus-user-agent-strings): How robots.txt tokens differ from user-agent strings: what you write in robots.txt, what arrives in the request, and tokens that never appear in logs. - [How to verify Web Bot Auth signatures | Agent Ready Crawlers](https://agentreadycrawlers.com/guides/verify-web-bot-auth): How to verify Web Bot Auth signatures: read Signature-Agent, fetch the key directory, check the RFC 9421 signature and keep IP checks as a fallback. - [What Content-Signals changes | Agent Ready Crawlers](https://agentreadycrawlers.com/guides/what-content-signals-change): What Content-Signals changes: the search, ai-input and ai-train lines in robots.txt, Cloudflare's defaults and why they enforce nothing. - [What pay per crawl pays and who is paying | Agent Ready Crawlers](https://agentreadycrawlers.com/guides/what-pay-per-crawl-pays): What pay per crawl pays and who is paying: Cloudflare, AWS WAF and TollBit compared, and why no payout totals are public yet. - [Which AI bots to allow, by type of site | Agent Ready Crawlers](https://agentreadycrawlers.com/guides/which-bots-to-allow-by-site-type): Which AI bots to allow for a publisher, a docs site, a store, a logged-in app or a closed site, with a robots.txt generated from the Crawler Index. ## Method - [Method: index, block rates, vendors | Agent Ready Crawlers](https://agentreadycrawlers.com/method): Method: how the Crawler Index, the Block Rate Report and the vendor rubric are built, what the crawl does, and why we recommend Ora as the access check. ## News Section index: https://agentreadycrawlers.com/news/llms.txt - [News: crawler and agent changes | Agent Ready Crawlers](https://agentreadycrawlers.com/news): 15 dated news items on AI crawlers, agents and bot management, from 2026-08-19 to 2026-11-02, each tied to the Crawler Index rows it changes. Read 2026-11-04. - [Google retires Google-NotebookLM string | Agent Ready Crawlers](https://agentreadycrawlers.com/news/2026-08-19-google-retires-the-google-notebooklm-user-agent): Google retires the Google-NotebookLM user agent. The page, last updated 2026-08-19, also lists Google-Agent and its experimental Web Bot Auth identity. - [Cloudflare opens BotBase for Operators | Agent Ready Crawlers](https://agentreadycrawlers.com/news/2026-08-28-cloudflare-opens-botbase-for-operators): Cloudflare opens BotBase for Operators. Cloudflare says accurate entries decide whether a bot earns and keeps Verified status. - [HUMAN publishes Agent Trust Levels | Agent Ready Crawlers](https://agentreadycrawlers.com/news/2026-09-02-human-publishes-agent-trust-levels): HUMAN publishes Agent Trust Levels. Customers set permissions per action such as account creation and checkout. Dated 2026-09-02. - [Meta's Muse agent does not identify itself | Agent Ready Crawlers](https://agentreadycrawlers.com/news/2026-09-08-meta-launches-muse-a-personal-agent-that-does-not-identify-i): Meta launches Muse, a personal agent that does not identify itself. Meta released Muse in the US on iOS, Android, the web and WhatsApp. - [Cloudflare adds Disallow AI Training | Agent Ready Crawlers](https://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-lab): Cloudflare adds Disallow AI Training and the Accountable label: a no-training line in robots.txt that keeps mixed-use crawlers in search. Dated 2026-09-15. - [Arcjet launches Agent Runtime Security | Agent Ready Crawlers](https://agentreadycrawlers.com/news/2026-09-17-arcjet-launches-agent-runtime-security): Arcjet launches Agent Runtime Security. Its inbound bot protection still verifies known bots by IP and reverse DNS. Dated 2026-09-17. - [DataDome joins Experian Agent Trust | Agent Ready Crawlers](https://agentreadycrawlers.com/news/2026-10-01-datadome-joins-experian-agent-trust): DataDome joins Experian Agent Trust. DataDome already lists Web Bot Auth, KYA tokens, IP lists and reverse DNS as identification methods. - [DataDome Q3: 25 billion agent requests | Agent Ready Crawlers](https://agentreadycrawlers.com/news/2026-10-15-datadome-q3-report-25-billion-ai-agent-requests): DataDome Q3 report: 25 billion AI agent requests. It counted 29.1 million Muse requests in under two weeks without self-identification. - [Anthropic refreshes its crawler IP list | Agent Ready Crawlers](https://agentreadycrawlers.com/news/2026-10-20-anthropic-refreshes-its-crawler-ip-list): Anthropic refreshes its crawler IP list. Anthropic warns that IP blocking may stop its bots reading robots.txt. Dated 2026-10-20. - [Claude Duet browses as Claude-Agent | Agent Ready Crawlers](https://agentreadycrawlers.com/news/2026-10-21-anthropic-s-claude-duet-starts-browsing-with-a-claude-agent): Anthropic's Claude Duet starts browsing with a Claude-Agent token. Claude Duet launched as a waitlist research preview for US Pro and Max subscribers. - [IETF proposes common IP-range format | Agent Ready Crawlers](https://agentreadycrawlers.com/news/2026-10-22-ietf-drafts-propose-a-common-ip-range-format-and-crawler-bes): IETF drafts propose a common IP-range format and crawler best practices: one for publishing IP ranges, one for crawler conduct. Dated 2026-10-22. - [OpenAI updates the ChatGPT-User IP list | Agent Ready Crawlers](https://agentreadycrawlers.com/news/2026-10-27-openai-updates-the-chatgpt-user-ip-list): OpenAI updates the ChatGPT-User IP list. openai.com/chatgpt-user.json carried creationTime 2026-10-27 when read on 2026-11-04. - [Vercel splits AI crawlers from AI assistants | Agent Ready Crawlers](https://agentreadycrawlers.com/news/2026-10-29-vercel-splits-ai-crawlers-from-ai-assistants): Vercel splits AI crawlers from AI assistants. Vercel verifies known bots by IP, reverse DNS and Web Bot Auth. Dated 2026-10-29. - [Web Bot Auth working-group draft reaches -01 | Agent Ready Crawlers](https://agentreadycrawlers.com/news/2026-10-29-web-bot-auth-working-group-draft-reaches-01): Web Bot Auth working-group draft reaches -01. A registry and Signature Agent card draft (-04) followed on 2026-10-27. Dated 2026-10-29. - [Signed-Agent-Metadata header proposed | Agent Ready Crawlers](https://agentreadycrawlers.com/news/2026-11-02-signed-agent-metadata-header-proposed-for-web-bot-auth): Signed-Agent-Metadata header proposed for Web Bot Auth. A new draft would carry signed metadata about an agent beside its signature. Dated 2026-11-02. ## Tools - [robots.txt builder for AI crawlers | Agent Ready Crawlers](https://agentreadycrawlers.com/robots-builder): Build a robots.txt from the Crawler Index: allow or refuse training, search, answer and agent tokens, then copy the file. ## Vendors Section index: https://agentreadycrawlers.com/vendors/llms.txt - [Vendors: operators, bot management, licensing | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors): The full vendor list: 73 crawler operators, bot management, verification and licensing vendors with standards, dated prices and sources. Read 2026-11-05. - [Akamai: bot management profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/akamai): Akamai: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 1 of 15. - [Alternatives to Akamai | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/akamai/alternatives): Alternatives to Akamai: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04. - [Amazon: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/amazon): Amazon: crawler operator profile covering Amazonbot, Amzn-SearchBot, Amzn-User and 1 more. Sourced facts, standards, dated prices and limits, read 2026-11-05. - [American Express: vendor profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/american-express): American Express: agent verification provider profile covering ACE developer kit and Amex Agent Purchase Protection. Read 2026-11-05. - [Anchor Browser: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/anchor): Anchor Browser: crawler operator profile covering Anchor cloud browsers, Web Unlocker and OmniConnect. Standards, dated prices and limits, read 2026-11-05. - [Anthropic: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/anthropic): Anthropic: crawler operator profile covering ClaudeBot, Claude-SearchBot, Claude-User and 1 more. Standards, dated prices and limits, read 2026-11-05. - [Apify: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/apify): Apify: crawler operator profile covering Website Content Crawler and Apify platform and Store. Standards, dated prices and limits, read 2026-11-05. - [Apple: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/apple): Apple: crawler operator profile covering Applebot, Applebot-Extended and iTMS. Sourced facts, standards, dated prices and limits, read 2026-11-05. - [Arcjet: bot management profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/arcjet): Arcjet: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 11 of 15. - [Alternatives to Arcjet | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/arcjet/alternatives): Alternatives to Arcjet: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04. - [Auth0 for AI Agents: bot management profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/auth0-ai-agents): Auth0 for AI Agents: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. - [Alternatives to Auth0 for AI Agents | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/auth0-ai-agents/alternatives): Alternatives to Auth0 for AI Agents: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. - [AWS WAF Bot Control: bot management profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/aws-waf-bot-control): AWS WAF Bot Control: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. - [Alternatives to AWS WAF Bot Control | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/aws-waf-bot-control/alternatives): Alternatives to AWS WAF Bot Control: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. - [Block: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/block): Block: crawler operator profile covering goose. Sourced facts, a standards table, dated prices and what it does not do, read 2026-11-05. - [Browser Use: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/browser-use): Browser Use: crawler operator profile covering Browser Infrastructure (cloud browsers), Browser Use Agents and browser-use (open source). Read 2026-11-05. - [Browserbase: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/browserbase): Browserbase: crawler operator profile covering Browserbase cloud browsers and Stagehand. Sourced facts, standards, dated prices and limits, read 2026-11-05. - [Bot management vendors: 23 vendor profiles | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/by-type/bot-management): Bot management vendors: 23 profiles with standards, dated prices, sources and what each does not do. 14 of 23 document support for Web Bot Auth. - [Crawler operators: 31 vendor profiles | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/by-type/crawler-operator): Crawler operators: 31 profiles with standards, dated prices, sources and what each does not do. 16 of 31 document support for RFC 9309. Read 2026-11-05. - [Licensing intermediaries: 8 vendor profiles | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/by-type/licensing-intermediary): Licensing intermediaries: 8 profiles with standards, dated prices, sources and what each does not do. 3 of 8 document support for RFC 9309. Read 2026-11-05. - [Verification providers: 11 vendor profiles | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/by-type/verification-provider): Verification providers: 11 profiles with standards, dated prices, sources and what each does not do. 3 of 11 document support for Web Bot Auth. - [Castle: bot management vendor profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/castle): Castle: bot management vendor profile covering Castle Risk API with AI agent identification. Standards, dated prices and limits, read 2026-11-05. - [Centinel Analytica: vendor profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/centinel-analytica): Centinel Analytica: bot management vendor profile covering Centinel bot management and Bot directory and AI crawler list. Read 2026-11-05. - [Cequence Security: vendor profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/cequence): Cequence Security: bot management vendor profile covering Bot Management with Agent Trust and AI Gateway. Standards, dated prices and limits, read 2026-11-05. - [Cloudflare: bot management profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/cloudflare): Cloudflare: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 1 of 15. - [Alternatives to Cloudflare | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/cloudflare/alternatives): Alternatives to Cloudflare: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04. - [Cohere: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/cohere): Cohere: crawler operator profile. Sourced facts, a standards table, dated prices and what it does not do, read 2026-11-05. - [Common Crawl: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/common-crawl): Common Crawl: crawler operator profile covering CCBot and Opt-Out Ledger. Sourced facts, standards, dated prices and limits, read 2026-11-05. - [Dappier: licensing intermediary profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/dappier): Dappier: licensing intermediary profile covering RAG API Marketplace, Agent storefront and Ask AI and WordPress plugin. Read 2026-11-05. - [DataDome: bot management profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/datadome): DataDome: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 1 of 15. - [Alternatives to DataDome | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/datadome/alternatives): Alternatives to DataDome: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04. - [Descope: agent verification provider profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/descope): Descope: agent verification provider profile covering Agentic Identity Hub. Sourced facts, standards, dated prices and limits, read 2026-11-05. - [Diffbot: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/diffbot): Diffbot: crawler operator profile covering Crawl (Crawlbot), Extract APIs and Knowledge Graph and Web Search. Sources, standards and limits, read 2026-11-05. - [DuckDuckGo: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/duckduckgo): DuckDuckGo: crawler operator profile covering DuckDuckBot and DuckAssistBot. Sourced facts, standards, dated prices and limits, read 2026-11-05. - [Exa: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/exa): Exa: crawler operator profile covering Exa Search, Contents and Answer APIs. Sourced facts, standards, dated prices and limits, read 2026-11-05. - [F5: bot management vendor profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/f5): F5: bot management vendor profile covering F5 Distributed Cloud Bot Defense. Sourced facts, standards, dated prices and limits, read 2026-11-05. - [Fastly: bot management profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/fastly): Fastly: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 10 of 15. - [Alternatives to Fastly | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/fastly/alternatives): Alternatives to Fastly: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04. - [FetchRight: licensing intermediary profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/fetchright): FetchRight: licensing intermediary profile covering Peek-Then-Pay and FetchRight platform. Sourced facts, standards, dated prices and limits, read 2026-11-05. - [Fingerprint: bot management vendor profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/fingerprint): Fingerprint: bot management vendor profile covering Bot Detection (Smart Signal) and Authorized AI Agent Detection. Read 2026-11-05. - [Firecrawl: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/firecrawl): Firecrawl: crawler operator profile covering Firecrawl API (scrape, crawl, map, search). Sourced facts, standards, dated prices and limits, read 2026-11-05. - [Google: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/google): Google: crawler operator profile covering Googlebot, Google-Extended, Google-Agent and 3 more. Standards, dated prices and limits, read 2026-11-05. - [Google Cloud Fraud Defense: vendor profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/google-cloud-fraud-defense): Google Cloud Fraud Defense: bot management vendor profile covering Google Cloud Fraud Defense (reCAPTCHA) and Google Cloud Armor. Read 2026-11-05. - [hCaptcha: bot management vendor profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/hcaptcha): hCaptcha: bot management vendor profile covering hCaptcha Enterprise with Agent Controls and hCaptcha Pro. Sources, standards and limits, read 2026-11-05. - [Hive: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/hive-imagesift): Hive: crawler operator profile covering ImageSift reverse image search and Deploy inference platform. Standards, dated prices and limits, read 2026-11-05. - [Huawei: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/huawei): Huawei: crawler operator profile covering PetalBot and Verify PetalBot. Sourced facts, standards, dated prices and limits, read 2026-11-05. - [HUMAN Security: bot management profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/human-security): HUMAN Security: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. - [Alternatives to HUMAN Security | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/human-security/alternatives): Alternatives to HUMAN Security: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. - [Hyperbrowser: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/hyperbrowser): Hyperbrowser: crawler operator profile covering Hyperbrowser sessions and Hyperbrowser Web API (fetch, search, extract). Read 2026-11-05. - [IAB Tech Lab: licensing intermediary profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/iab-tech-lab): IAB Tech Lab: licensing intermediary profile covering CoMP 1.0, Spiders and Bots List and Agent Registry. Standards, dated prices and limits, read 2026-11-05. - [Imperva: bot management profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/imperva): Imperva: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 11 of 15. - [Alternatives to Imperva | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/imperva/alternatives): Alternatives to Imperva: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04. - [Kasada: bot management profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/kasada): Kasada: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 9 of 15. - [Alternatives to Kasada | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/kasada/alternatives): Alternatives to Kasada: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04. - [Kernel: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/kernel): Kernel: crawler operator profile covering Kernel cloud browsers and Kernel Search (KernelSearchBot). Standards, dated prices and limits, read 2026-11-05. - [Known Agents: bot management vendor profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/known-agents): Known Agents: bot management vendor profile covering Agent Analytics, Automatic Robots.txt, Bad Bot Blocking and 1 more. Read 2026-11-05. - [Link11: bot management vendor profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/link11): Link11: bot management vendor profile covering Link11 WAAP (WAF, API Security, Bot Management, Web DDoS). Standards, dated prices and limits, read 2026-11-05. - [Manus: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/manus): Manus: crawler operator profile covering Manus Cloud Browser and Manus Browser Operator. Sourced facts, standards, dated prices and limits, read 2026-11-05. - [Mastercard: vendor profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/mastercard): Mastercard: agent verification provider profile covering Mastercard Agent Pay and Developer Playbook on Preparing for Agentic Commerce. Read 2026-11-05. - [Meta: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/meta): Meta: crawler operator profile covering Meta-ExternalAgent, Meta-WebIndexer, Meta-ExternalFetcher and 2 more. Sources, standards and limits, read 2026-11-05. - [Microsoft: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/microsoft): Microsoft: crawler operator profile covering Bingbot, AI Training and AI Grounding preferences and Publisher Content Marketplace. Read 2026-11-05. - [Microsoft Entra Agent ID: vendor profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/microsoft-entra-agent-id): Microsoft Entra Agent ID: agent verification provider profile covering Microsoft Entra Agent ID. Standards, dated prices and limits, read 2026-11-05. - [Mistral AI: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/mistral-ai): Mistral AI: crawler operator profile covering MistralAI-User, MistralAI-Index and MistralAI-Training. Standards, dated prices and limits, read 2026-11-05. - [Netacea: bot management vendor profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/netacea): Netacea: bot management vendor profile covering Netacea Bot Protection. Sourced facts, standards, dated prices and limits, read 2026-11-05. - [Netlify: bot management profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/netlify): Netlify: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 15 of 15. - [Alternatives to Netlify | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/netlify/alternatives): Alternatives to Netlify: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04. - [Notte: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/notte): Notte: crawler operator profile covering Notte browser sessions and agents. Sourced facts, standards, dated prices and limits, read 2026-11-05. - [OpenAI: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/openai): OpenAI: crawler operator profile covering GPTBot, OAI-SearchBot, OAI-AdsBot and 2 more. Sourced facts, standards, dated prices and limits, read 2026-11-05. - [Parallel Web Systems: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/parallel): Parallel Web Systems: crawler operator profile covering Parallel Search, Extract and Task APIs. Standards, dated prices and limits, read 2026-11-05. - [Perplexity: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/perplexity): Perplexity: crawler operator profile covering PerplexityBot and Perplexity-User. Sourced facts, standards, dated prices and limits, read 2026-11-05. - [ProRata: licensing intermediary profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/prorata): ProRata: licensing intermediary profile covering ProRata Attribution, Gist Answers and Gist GEO. Standards, dated prices and limits, read 2026-11-05. - [Prove: agent verification provider profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/prove): Prove: agent verification provider profile covering Prove Verified Agent, Prove Agent Pay and Prove Agent Protect. Read 2026-11-05. - [Radware: bot management vendor profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/radware): Radware: bot management vendor profile covering Radware Bot Manager and Radware Agent Trust Management. Standards, dated prices and limits, read 2026-11-05. - [Vendor ranking and rubric | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/ranking): Ranking of 15 bot management and agent identity vendors on a published rubric: verification, per-purpose policy, charging, commerce agents and reporting. - [RSL Collective: licensing intermediary profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/rsl-collective): RSL Collective: licensing intermediary profile covering RSL 1.0 standard and RSL Collective licensing platform. Read 2026-11-05. - [Skyfire: bot management profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/skyfire): Skyfire: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 6 of 15. - [Alternatives to Skyfire | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/skyfire/alternatives): Alternatives to Skyfire: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04. - [Stytch IsAgent: bot management profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/stytch-isagent): Stytch IsAgent: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. - [Alternatives to Stytch IsAgent | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/stytch-isagent/alternatives): Alternatives to Stytch IsAgent: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. - [Supertab: licensing intermediary profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/supertab): Supertab: licensing intermediary profile covering Supertab Connect and Supertab Pass. Sourced facts, standards, dated prices and limits, read 2026-11-05. - [Timpi: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/timpi): Timpi: crawler operator profile covering Timpi Search, Timpi Data and Timpi DePIN nodes. Sourced facts, standards, dated prices and limits, read 2026-11-05. - [TollBit: bot management profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/tollbit): TollBit: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 6 of 15. - [Alternatives to TollBit | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/tollbit/alternatives): Alternatives to TollBit: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04. - [Trulioo: agent verification provider profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/trulioo): Trulioo: agent verification provider profile covering Know Your Agent (KYA). Sourced facts, standards, dated prices and limits, read 2026-11-05. - [Vercel: bot management profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/vercel-botid): Vercel: bot management and agent identity profile with a graded rubric, price read 2026-11-04, what it does not do, dated news and sources. Rank 6 of 15. - [Alternatives to Vercel | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/vercel-botid/alternatives): Alternatives to Vercel: 14 bot management and agent identity vendors on one rubric, with the sourced difference for each head-to-head. Read 2026-11-04. - [Verisoul: bot management vendor profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/verisoul): Verisoul: bot management vendor profile covering Agent & Bot Detection and Agent Atlas. Sourced facts, standards, dated prices and limits, read 2026-11-05. - [Visa: agent verification provider profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/visa): Visa: agent verification provider profile covering Trusted Agent Protocol, Visa Intelligent Commerce and Agentic Directory. Read 2026-11-05. - [Vouched: agent verification provider profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/vouched): Vouched: agent verification provider profile covering Know Your Agent: Detect, Know Your Agent: Enforce and Govern, KnowThat.ai and 1 more. Read 2026-11-05. - [Webz.io: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/webz-io): Webz.io: crawler operator profile covering Open Web APIs (News, Blogs, Forums, Reviews, Search) and Dark Web API and Breach API. Read 2026-11-05. - [World: agent verification provider profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/world): World: agent verification provider profile covering AgentKit (beta) and World ID. Sourced facts, standards, dated prices and limits, read 2026-11-05. - [xAI: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/xai): xAI: crawler operator profile covering Grok Bot and Web Search tool (xAI API). Sourced facts, standards, dated prices and limits, read 2026-11-05. - [You.com: crawler operator profile | Agent Ready Crawlers](https://agentreadycrawlers.com/vendors/you-com): You.com: crawler operator profile covering YouBot. Sourced facts, a standards table, dated prices and what it does not do, read 2026-11-05. ## Verification - [Verification Ledger: who publishes what | Agent Ready Crawlers](https://agentreadycrawlers.com/verification): The Verification Ledger: which of 31 operators publish IP lists, reverse DNS patterns or Web Bot Auth key directories, with URL and date read. - [Verified bot programmes compared | Agent Ready Crawlers](https://agentreadycrawlers.com/verification/programmes-compared): IP lists, reverse DNS, Web Bot Auth, Cloudflare signed agents, Visa TAP and Mastercard Agent Pay compared: what a site checks and which Index rows use each.