How to test what a 403 does to an AI answer about you

In short

Request your own pages with each crawler token and with a plain fetch, compare status codes, then ask the assistant about your site and see whether it quotes you. OpenAI says sites opted out of OAI-SearchBot do not appear in ChatGPT search answers, though they can appear as navigational links.

A 403 to an agent is invisible in your analytics and decisive for the answer. The test takes an afternoon and needs no tool.

In practice

1. Request your own key pages once with a plain fetch and once with each crawler's user-agent string from the Index, and compare the status codes. Ora counts a 401, 402, 403, 451 or bot challenge to an agent, while a normal browser gets in, as blocked.
2. Ask the assistant about your site and see whether it quotes you. Ora's lab found (2026-10-19) that serving 403 to agent user agents left both a plain-fetch and a JavaScript client at 0 of 5 retrieved answers on one docs site; that is a vendor finding.
3. Read the result against what operators say. Sites opted out of OAI-SearchBot do not appear in ChatGPT search answers, though they can appear as navigational links. Disabling Claude-User prevents retrieval in response to user queries.
4. Remember Perplexity-User generally ignores robots.txt, so a 403 at the WAF is the only refusal it sees.
5. Compare with the field. Cloudflare reports that less than 1 percent of its sites block search bots while 17 percent block training. The crawl behind this site found 12.9 percent of home pages answering 401 or 403 to its labelled user agent.

What the sources say

• Vendor claim Ora's lab found (2026-10-19) that serving 403 to agent user agents left both a plain-fetch client and a JavaScript client at 0 of 5 retrieved answers, across 190 probes of one docs site in 19 configurations. Source, read 2026-11-04.
• Vendor claim Ora's lab found (2026-10-08) that on sites that are not agent-ready, agent runs get blocked by the site 2.1 times as often and 16 percent of answers admit they could not access the information. Source, read 2026-11-04.
• Primary source OpenAI says sites opted out of OAI-SearchBot do not appear in ChatGPT search answers, though they can appear as navigational links. Source, read 2026-11-04.
• Primary source Anthropic says disabling Claude-User prevents retrieval in response to user queries. Source, read 2026-11-04.
• Primary source Perplexity-User generally ignores robots.txt, so a 403 at the WAF is the only refusal it sees. Source, read 2026-11-04.
• Primary source Cloudflare reports less than 1 percent of its sites block search bots while 17 percent block training. Source, read 2026-11-04.

Related

• The Block Rate Report
• Perplexity-User profile
• OAI-SearchBot profile
• How Ora's bot-detection check works
• 403 to AI agents: what the crawl and Ora show
• How to test whether AI agents can reach your site
• 401 and 403 by edge and bot-product header

Last reviewed 2026-11-04