--- title: "Live answer fetch crawlers and agents | Agent Ready Crawlers" description: "Live answer fetch in the Crawler Index: 8 AI crawlers and agents from 8 operators, with tokens, verification and block rates. Read 2026-11-04." canonical: https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch site: "Agent Ready Crawlers" last_updated: 2026-11-04 section: "Crawlers" --- # Live answer fetch crawlers and agents | Agent Ready Crawlers > Live answer fetch in the Crawler Index: 8 AI crawlers and agents from 8 operators, with tokens, verification and block rates. Read 2026-11-04. Source: https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # Live answer fetch: 8 rows In short 8 rows in the Index are filed under live answer fetch, from 8 operators. 1 publish nothing a site can verify and 1 have no robots.txt token. Of the 6 tokens the crawl measured, ChatGPT-User is refused by 9.7 percent of top-10,000 sites and MistralAI-User by 7.7 percent. - [Training](https://agentreadycrawlers.com/crawlers/purpose/training/) - [Search index](https://agentreadycrawlers.com/crawlers/purpose/search-index/) - [Live answer fetch](https://agentreadycrawlers.com/crawlers/purpose/live-answer-fetch/) - [Task agent](https://agentreadycrawlers.com/crawlers/purpose/task-agent/) - [Shopping agent](https://agentreadycrawlers.com/crawlers/purpose/shopping-agent/) - [Other](https://agentreadycrawlers.com/crawlers/purpose/other/) - [All rows](https://agentreadycrawlers.com/crawlers/) Fetches a page because a person asked an assistant about it, one request at a time. Live answer fetch: 8 rows. | Operator | Token | Purpose | Respects robots.txt | Verification | Block rate 2026-11-04 | Profile | | --- | --- | --- | --- | --- | --- | --- | | OpenAI | ChatGPT-User | Live answer fetch | May not apply | IP list | 9.7% | Profile | | Anthropic | Claude-User | Live answer fetch | Yes | IP list | 8.1% | Profile | | Perplexity | Perplexity-User | Live answer fetch | May not apply | IP list | 8.3% | Profile | | Google | Google-GeminiNotebook (was Google-NotebookLM) no token | Live answer fetch | May not apply | IP list | no token | Profile | | Amazon | Amzn-User | Live answer fetch | May not apply | IP list | not measured | Profile | | Meta | meta-externalfetcher | Live answer fetch | May not apply | None published | 8.5% | Profile | | DuckDuckGo | DuckAssistBot | Live answer fetch | Yes | IP list | 8.1% | Profile | | Mistral AI | MistralAI-User | Live answer fetch | Yes | IP list | 7.7% | Profile | ## Block rates in this purpose Share of the 5,938 top-10,000 sites with a parsed robots.txt that disallow the token at the root, crawl of 2026-11-04. "Named" counts sites with a group for the token; "via \*" counts sites that refuse it only because their wildcard group refuses everything. | Token | Refused | Named | Via * | All crawled | | --- | --- | --- | --- | --- | | ChatGPT-User | 9.7% | 6.3% | 3.3% | 6.9% | | Meta-ExternalFetcher | 8.5% | 4.7% | 3.8% | 6.0% | | Perplexity-User | 8.3% | 4.6% | 3.7% | 5.9% | | Claude-User | 8.1% | 4.5% | 3.6% | 5.7% | | DuckAssistBot | 8.1% | 4.3% | 3.8% | 5.8% | | MistralAI-User | 7.7% | 3.8% | 3.9% | 5.5% | ## Sources - [Purpose definitions (this site)](https://agentreadycrawlers.com/method/) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 - [Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) read 2026-11-04 - [Google user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) read 2026-11-04 - [Amazon Web Search crawlers](https://developer.amazon.com/amazonbot) read 2026-11-04 - [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) read 2026-11-04 Last reviewed 2026-11-04