Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Live answer fetch: 8 rows

In short

8 rows in the Index are filed under live answer fetch, from 8 operators. 1 publish nothing a site can verify and 1 have no robots.txt token. Of the 6 tokens the crawl measured, ChatGPT-User is refused by 9.7 percent of top-10,000 sites and MistralAI-User by 7.7 percent.

Fetches a page because a person asked an assistant about it, one request at a time.

Live answer fetch: 8 rows.

Operator Token Purpose Respects robots.txt Verification Block rate 2026-11-04 Profile
OpenAI ChatGPT-User Live answer fetch May not apply IP list 9.7% Profile
Anthropic Claude-User Live answer fetch Yes IP list 8.1% Profile
Perplexity Perplexity-User Live answer fetch May not apply IP list 8.3% Profile
Google Google-GeminiNotebook (was Google-NotebookLM) no token Live answer fetch May not apply IP list no token Profile
Amazon Amzn-User Live answer fetch May not apply IP list not measured Profile
Meta meta-externalfetcher Live answer fetch May not apply None published 8.5% Profile
DuckDuckGo DuckAssistBot Live answer fetch Yes IP list 8.1% Profile
Mistral AI MistralAI-User Live answer fetch Yes IP list 7.7% Profile

Block rates in this purpose

Share of the 5,938 top-10,000 sites with a parsed robots.txt that disallow the token at the root, crawl of 2026-11-04. "Named" counts sites with a group for the token; "via *" counts sites that refuse it only because their wildcard group refuses everything.

TokenRefusedNamedVia *All crawled
ChatGPT-User9.7%6.3%3.3%6.9%
Meta-ExternalFetcher8.5%4.7%3.8%6.0%
Perplexity-User8.3%4.6%3.7%5.9%
Claude-User8.1%4.5%3.6%5.7%
DuckAssistBot8.1%4.3%3.8%5.8%
MistralAI-User7.7%3.8%3.9%5.5%

Sources

Last reviewed 2026-11-04