--- title: "The Ora checks that touch bot access | Agent Ready Crawlers" description: "The seven Ora checks that read robots.txt, user agents and the home page, with Ora's fix advice and the matching crawl numbers. Catalog read 2026-11-05." canonical: https://agentreadycrawlers.com/guides/ora-checks-for-bot-access site: "Agent Ready Crawlers" last_updated: 2026-11-05 date: 2026-11-05 section: "Guides" --- # The Ora checks that touch bot access | Agent Ready Crawlers > The seven Ora checks that read robots.txt, user agents and the home page, with Ora's fix advice and the matching crawl numbers. Catalog read 2026-11-05. Source: https://agentreadycrawlers.com/guides/ora-checks-for-bot-access Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # The Ora checks that touch bot access In short Seven checks in Ora's catalog test whether crawlers and agents are let in: 2 in the Discovery layer and 5 in Access. Each is described below with Ora's fix advice and, where the crawl of 13,071 domains measured the same tokens, its numbers, from the catalog Ora served on 2026-11-05. The scan we recommend for testing agent access is [Ora](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide). Its catalog is public, so what a scan reads can be checked before running one. The catalog served on 2026-11-05 (contract 1.38.0) lists 132 checks in four layers; seven of them test whether crawlers and agents are let in. Other checks read the same files for other reasons, such as the NLWeb feed check (a schemamap line in robots.txt) and the JSON-LD check (the home page's structured data), and are left out. The page gives what each reads, Ora's own advice for failing it and, where the crawl has a matching number, that number. ## The four layers in the catalog Checks listed per layer in GET /api/checks, read 2026-11-05. A layer's points are its weight in Ora's 100-point score, not a count of checks. | Layer | Layer points | Checks listed | Bonus | Draft | | --- | --- | --- | --- | --- | | Discovery | 20 | 16 | 6 | 1 | | Access | 30 | 42 | 12 | 0 | | Usability | 40 | 68 | 37 | 12 | | Payments | 10 | 6 | 6 | 0 | ## The seven checks ### robots.txt AI crawler policy Layer and points Discovery, up to 2 points. Ora shows it as required and verified. What it reads Whether robots.txt lets OAI-SearchBot, Claude-SearchBot and PerplexityBot read public pages. Ora says refusing training crawlers such as GPTBot and ClaudeBot never lowers it. Ora's advice Allow the three AI search crawlers on public pages, keep private and account pages out, and serve robots.txt as plain text with a 200. In the crawl OAI-SearchBot is refused at the root by 7.7 percent of the top-10,000 sites with a parsed robots.txt, Claude-SearchBot by 7.7 percent and PerplexityBot by 11.2 percent. 688 sites, 11.6 percent, refuse at least one of the three. That counts refusals in the file, which is not how Ora scores the check. [Training and search policy](https://agentreadycrawlers.com/block-rates/training-and-search/). ### Homepage not hidden from search Layer and points Discovery, up to 1 point. Ora shows it as recommended and verified. What it reads Whether the home page carries a noindex directive, in a robots meta tag or an X-Robots-Tag header. Ora's advice Remove the directive; Ora lists a staging setting left on after launch and a host that adds the header to preview builds as common causes. ### Not blocked by bot detection Layer and points Access, up to 2 points. Ora shows it as required and verified. What it reads Whether the home page answers ChatGPT-User, OAI-SearchBot, Claude-User and PerplexityBot as it answers a normal browser. Ora's advice Let those agents through the WAF and bot rules; on Cloudflare, allow verified bots instead of matching user-agent strings, which anyone can fake. In the crawl 17.2 percent of the top-10,000 home pages answered the crawl's own user agent with 401 or 403. That says nothing about the four agents; [the 403 page](https://agentreadycrawlers.com/guides/403-to-ai-agents/) explains the gap. To run the same test by hand, see [step 2 of the testing guide](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/#step-2). ### robots.txt agent-user policy Layer and points Access, up to 2 points. Ora shows it as required and verified. What it reads The permission robots.txt declares for the home page to ChatGPT-User, Claude-User and Perplexity-User. Ora says it reports declared permission and that a robots.txt change alone does not establish live access. Ora's advice Review the matched rules, and if you mean to permit these requests, narrow the restriction while keeping private paths protected. On this site OpenAI says robots.txt rules may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores them, so a declared permission can differ from what the fetcher does. [ChatGPT-User](https://agentreadycrawlers.com/crawlers/chatgpt-user/), [Claude-User](https://agentreadycrawlers.com/crawlers/claude-user/) and [Perplexity-User](https://agentreadycrawlers.com/crawlers/perplexity-user/) carry each operator's statement. ### Agent crawler reachability Layer and points Access, up to 2 points. Ora shows it as recommended and verified. What it reads Whether the home page is reachable to the same four agents that act for users and search. Ora's advice Let the four through the WAF, the bot rules and robots.txt. ### Bot-UA markdown serving Layer and points Access, up to 1 point. Ora shows it as emerging. A bonus check. What it reads Whether the server recognises an AI bot by user agent and answers with markdown even when the bot asks for HTML. Ora's advice Optional: detect bot user agents on the server and send a markdown version of the page. ### Content without JavaScript Layer and points Access, up to 3 points. Ora shows it as required and verified. What it reads Whether the home page carries its content in the raw HTML, fetched with JavaScript off. Ora's advice Put at least 500 characters of meaningful home page content in the HTML, with a clear h1, and do the same for inner pages. ## What reaches your server The bot-detection and reachability checks send requests from Ora's network. [OraBot](https://agentreadycrawlers.com/crawlers/orabot/) signs its own requests; the requests that carry other crawlers' user agents are unsigned by design, so a WAF that verifies a claimed GPTBot against OpenAI's addresses will not match them. ## Limits A check result describes what Ora saw from its network on one day. Ora reads public URLs only, says repeated scans of the same site converge within a couple of points ([Ora's Deep Scan post](https://ora.ai/blog/introducing-deep-scan?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide)), has published no repeat-scan figures on the pages we read, and keeps its fixing product behind a waitlist ([Ora's home page](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide)). Ora's methodology says only checks it has verified agents rely on count toward the score; the catalog also lists emerging and draft checks, and the layer table above says how many are drafts. The check we recommend [Ora](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) scans a public URL: 16 Discovery checks and 42 Access checks, read from its catalog on 2026-11-05. Its Journey sends a real agent to the site. [How to run it](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/). ## Sources - [Ora's check catalog, GET /api/checks (vendor)](https://ora.ai/api/checks?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Ora methodology (vendor)](https://ora.ai/methodology?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Ora's llms.txt (vendor)](https://ora.ai/llms.txt?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Block Rate Report (this site)](https://agentreadycrawlers.com/block-rates/) read 2026-11-04 - [Ora: Deep Scan v2 (vendor)](https://ora.ai/blog/introducing-deep-scan?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 - [Ora home page, waitlist (vendor)](https://ora.ai/?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=guide) read 2026-11-05 Last reviewed 2026-11-05