--- title: "What a 403 does to an AI answer | Agent Ready Crawlers" description: "How to test what a 403 does to an AI answer about you: fetch your pages as each crawler, compare status codes, then ask the assistant." canonical: https://agentreadycrawlers.com/guides/test-what-a-403-does site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-11-04 section: "Guides" --- # What a 403 does to an AI answer | Agent Ready Crawlers > How to test what a 403 does to an AI answer about you: fetch your pages as each crawler, compare status codes, then ask the assistant. Source: https://agentreadycrawlers.com/guides/test-what-a-403-does Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # How to test what a 403 does to an AI answer about you In short Request your own pages with each crawler token and with a plain fetch, compare status codes, then ask the assistant about your site and see whether it quotes you. OpenAI says sites opted out of OAI-SearchBot do not appear in ChatGPT search answers, though they can appear as navigational links. A 403 to an agent is invisible in your analytics and decisive for the answer. The test takes an afternoon and needs no tool. ## In practice 1. Request your own key pages once with a plain fetch and once with each crawler's user-agent string from the Index, and compare the status codes. Ora counts a 401, 402, 403, 451 or bot challenge to an agent, while a normal browser gets in, as blocked. 2. Ask the assistant about your site and see whether it quotes you. Ora's lab found (2026-10-19) that serving 403 to agent user agents left both a plain-fetch and a JavaScript client at 0 of 5 retrieved answers on one docs site; that is a vendor finding. 3. Read the result against what operators say. Sites opted out of OAI-SearchBot do not appear in ChatGPT search answers, though they can appear as navigational links. Disabling Claude-User prevents retrieval in response to user queries. 4. Remember Perplexity-User generally ignores robots.txt, so a 403 at the WAF is the only refusal it sees. 5. Compare with the field. Cloudflare reports that less than 1 percent of its sites block search bots while 17 percent block training. The crawl behind this site found 12.9 percent of home pages answering 401 or 403 to its labelled user agent. ## What the sources say - Vendor claim Ora's lab found (2026-10-19) that serving 403 to agent user agents left both a plain-fetch client and a JavaScript client at 0 of 5 retrieved answers, across 190 probes of one docs site in 19 configurations. [Source](https://ora.ai/blog/agentready-launch?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide), read 2026-11-04. - Vendor claim Ora's lab found (2026-10-08) that on sites that are not agent-ready, agent runs get blocked by the site 2.1 times as often and 16 percent of answers admit they could not access the information. [Source](https://ora.ai/blog/ax-is-the-new-aeo?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide), read 2026-11-04. - Primary source OpenAI says sites opted out of OAI-SearchBot do not appear in ChatGPT search answers, though they can appear as navigational links. [Source](https://platform.openai.com/docs/bots), read 2026-11-04. - Primary source Anthropic says disabling Claude-User prevents retrieval in response to user queries. [Source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), read 2026-11-04. - Primary source Perplexity-User generally ignores robots.txt, so a 403 at the WAF is the only refusal it sees. [Source](https://docs.perplexity.ai/guides/bots), read 2026-11-04. - Primary source Cloudflare reports less than 1 percent of its sites block search bots while 17 percent block training. [Source](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 2026-11-04. ## Related - [The Block Rate Report](https://agentreadycrawlers.com/block-rates/) - [Perplexity-User profile](https://agentreadycrawlers.com/crawlers/perplexity-user/) - [OAI-SearchBot profile](https://agentreadycrawlers.com/crawlers/oai-searchbot/) - [How Ora's bot-detection check works](https://agentreadycrawlers.com/method/#ora-on-this-layer) - [403 to AI agents: what the crawl and Ora show](https://agentreadycrawlers.com/guides/403-to-ai-agents/) - [How to test whether AI agents can reach your site](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/) - [401 and 403 by edge and bot-product header](https://agentreadycrawlers.com/block-rates/edge-and-bot-headers/) ## Sources - [ora.ai: agentready launch](https://ora.ai/blog/agentready-launch?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-04 - [ora.ai: ax is the new aeo](https://ora.ai/blog/ax-is-the-new-aeo?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-04 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04 - [Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 Last reviewed 2026-11-04