--- title: "403 to AI agents: what the crawl and Ora show | Agent Ready Crawlers" description: "What a 403 to an AI agent does, per Ora's lab, how many top home pages answer 401 or 403 in our crawl, and how to find your own refusals. Read 2026-11-05." canonical: https://agentreadycrawlers.com/guides/403-to-ai-agents site: "Agent Ready Crawlers" last_updated: 2026-11-05 date: 2026-11-05 section: "Guides" --- # 403 to AI agents: what the crawl and Ora show | Agent Ready Crawlers > What a 403 to an AI agent does, per Ora's lab, how many top home pages answer 401 or 403 in our crawl, and how to find your own refusals. Read 2026-11-05. Source: https://agentreadycrawlers.com/guides/403-to-ai-agents Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # 403 to AI agents In short Ora's lab found (2026-10-19) that returning 403 to agent user agents left both test clients at 0 of 5 retrieved answers; that is a vendor finding on one docs site. In our crawl 17.2 percent of top-10,000 home pages answered 401 or 403 to a labelled non-agent user agent, which says how common refusals are, not how many reach agents. ## What Ora's lab found Ora's research lab served one docs site under 19 configurations and asked two clients to pull a planted answer from it: a plain fetch, like most agent tools, and a client that runs JavaScript. It made 190 probes, five topics per client per configuration. Only two kinds of change stopped retrieval (Ora, 19 October 2026; a vendor finding, read 2026-11-05). | Site configuration | Plain fetch | Runs JavaScript | | --- | --- | --- | | Answer only in JavaScript (1 configuration) | 0 of 5 | 5 of 5 | | 403 to agent user agents (2 configurations) | 0 of 5 | 0 of 5 | | Every other configuration (16, including llms.txt, JSON-LD, sitemaps and markdown mirrors) | 5 of 5 | 5 of 5 | The same post reports a second result: across 1,056 business sites, answers written by an agent that could not read the site were 3.7 times as likely to miss every fact the user asked for, and agents clearly recommended agent-ready businesses almost twice as often. Both are Ora's claims about its own sample. Nothing here tests them, and a docs site in a lab is not your site. ## How common a refusal is in the crawl The crawl fetched 10,565 home pages with its own labelled user agent. 12.9 percent answered 401 or 403, 17.2 percent of the 7,545 fetched in the top 10,000 (13.0 percent in ranks 1 to 1,000 and 17.6 percent in ranks 1,001 to 10,000). 644 home pages, 6.1 percent, carried a Cloudflare challenge header. Home pages that answered 401 or 403 to the crawl user agent, crawl of 2026-11-04. Categories with at least 30 home pages fetched; the eight highest shown. | Group | 401 or 403 share | | --- | --- | | All crawled | 12.9% | | Top 10,000 | 17.2% | | News & Intelligence | 25.7% | | Gaming & Media | 12.9% | | CRM | 7.1% | | HR & Recruiting | 5.3% | | Communication & Support | 4.7% | | Education | 4.3% | | Marketing | 4.1% | | Consumer | 4.0% | Read the share as the size of the problem space and no more. The crawl identifies itself as `agent-readiness-crawl`, not as ChatGPT-User or PerplexityBot, so a site that refuses it may admit the agents, and a site that admits it may refuse them. The crawl also skipped 259 domains whose robots.txt disallowed it, and those are in no share above. ## Where the 403 is the refusal that counts For fetchers that a person triggers, operators say robots.txt may not decide whether the request is made. For these 7 rows, whatever your server answers is the whole policy. | Fetcher | Operator | What the operator says | Verification | | --- | --- | --- | --- | | ChatGPT-User | OpenAI | because these actions are initiated by a user, robots.txt rules may not apply. | IP list | | Perplexity-User | Perplexity | since a user requested the fetch, this fetcher generally ignores robots.txt rules. | IP list | | Google-Agent | Google | Google lists it as a user-triggered fetcher; these fetchers generally ignore robots.txt because the fetch was requested by a user. | Web Bot Auth | | Google-GeminiNotebook | Google | generally ignores robots.txt, per Google. | IP list | | Amzn-User | Amazon | because actions can be initiated by a user, Amzn-User may not follow all robots.txt directives. | IP list | | Meta-ExternalFetcher | Meta | may bypass robots.txt because it performs fetches requested by the user. | None published | | FacebookExternalHit | Meta | might bypass robots.txt when performing security or integrity checks. | None published | How each row's robots.txt statement is read is on [the method page](https://agentreadycrawlers.com/method/#index-method). ## Find your own refusals Start from the log, since it shows what real agents received. In the common combined log format the status is the ninth field, so this lists refusals to the agents named in Ora's test: ``` awk '$9 == 403 || $9 == 401' access.log | grep -E 'ChatGPT-User|OAI-SearchBot|Claude-User|PerplexityBot' ``` Then reproduce one with the commands in [step 2 of the testing guide](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/#step-2). Ora's bot-detection check does the same from its network and counts a 401, 402, 403, 451 or a bot challenge as blocked when a normal browser visit gets in (Ora's methodology, read 2026-11-05). ## What to change Allow by verified identity instead of by string. All four agents in Ora's test, ChatGPT-User, OAI-SearchBot, Claude-User and PerplexityBot, publish an IP list in the Index, so an address rule is possible; Ora's own advice for Cloudflare sites is to allow verified bots, since a user-agent match can be faked. After the change, repeat the test and the scan. [The Ora checks that touch bot access](https://agentreadycrawlers.com/guides/ora-checks-for-bot-access/) lists the four checks involved. The check we recommend [Ora](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) scans a public URL: 16 Discovery checks and 42 Access checks, read from its catalog on 2026-11-05. Its Journey sends a real agent to the site. [How to run it](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/). **Limits.** Ora reads public URLs only, and what it records as blocked comes from its own network. It says the same scan run twice converges within a couple of points ([Ora's Deep Scan post](https://ora.ai/blog/introducing-deep-scan?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide)), and the pages we read give no repeat-scan figures. Its fixing product is behind a waitlist ([Ora's home page](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide)). ## Sources - [Ora's lab: AgentReady launch, 19 Oct 2026 (vendor)](https://ora.ai/blog/agentready-launch?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Ora methodology (vendor)](https://ora.ai/methodology?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Ora's check catalog (vendor)](https://ora.ai/api/checks?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) read 2026-11-04 - [Block Rate Report, edition 2026-11-04 (this site)](https://agentreadycrawlers.com/block-rates/2026-11-04/) read 2026-11-04 - [Ora: Deep Scan v2 (vendor)](https://ora.ai/blog/introducing-deep-scan?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Ora home page, waitlist (vendor)](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 Last reviewed 2026-11-05