The Ora checks that touch bot access

In short

Seven checks in Ora's catalog test whether crawlers and agents are let in: 2 in the Discovery layer and 5 in Access. Each is described below with Ora's fix advice and, where the crawl of 13,071 domains measured the same tokens, its numbers, from the catalog Ora served on 2026-11-05.

The scan we recommend for testing agent access is Ora. Its catalog is public, so what a scan reads can be checked before running one. The catalog served on 2026-11-05 (contract 1.38.0) lists 132 checks in four layers; seven of them test whether crawlers and agents are let in. Other checks read the same files for other reasons, such as the NLWeb feed check (a schemamap line in robots.txt) and the JSON-LD check (the home page's structured data), and are left out. The page gives what each reads, Ora's own advice for failing it and, where the crawl has a matching number, that number.

The four layers in the catalog

The seven checks

robots.txt AI crawler policy

Layer and points

Discovery, up to 2 points. Ora shows it as required and verified.

What it reads

Whether robots.txt lets OAI-SearchBot, Claude-SearchBot and PerplexityBot read public pages. Ora says refusing training crawlers such as GPTBot and ClaudeBot never lowers it.

Ora's advice

Allow the three AI search crawlers on public pages, keep private and account pages out, and serve robots.txt as plain text with a 200.

In the crawl

OAI-SearchBot is refused at the root by 7.7 percent of the top-10,000 sites with a parsed robots.txt, Claude-SearchBot by 7.7 percent and PerplexityBot by 11.2 percent. 688 sites, 11.6 percent, refuse at least one of the three. That counts refusals in the file, which is not how Ora scores the check. Training and search policy.

Homepage not hidden from search

Layer and points

Discovery, up to 1 point. Ora shows it as recommended and verified.

What it reads

Whether the home page carries a noindex directive, in a robots meta tag or an X-Robots-Tag header.

Ora's advice

Remove the directive; Ora lists a staging setting left on after launch and a host that adds the header to preview builds as common causes.

Not blocked by bot detection

Layer and points

Access, up to 2 points. Ora shows it as required and verified.

What it reads

Whether the home page answers ChatGPT-User, OAI-SearchBot, Claude-User and PerplexityBot as it answers a normal browser.

Ora's advice

Let those agents through the WAF and bot rules; on Cloudflare, allow verified bots instead of matching user-agent strings, which anyone can fake.

In the crawl

17.2 percent of the top-10,000 home pages answered the crawl's own user agent with 401 or 403. That says nothing about the four agents; the 403 page explains the gap. To run the same test by hand, see step 2 of the testing guide.

robots.txt agent-user policy

Layer and points

Access, up to 2 points. Ora shows it as required and verified.

What it reads

The permission robots.txt declares for the home page to ChatGPT-User, Claude-User and Perplexity-User. Ora says it reports declared permission and that a robots.txt change alone does not establish live access.

Ora's advice

Review the matched rules, and if you mean to permit these requests, narrow the restriction while keeping private paths protected.

On this site

OpenAI says robots.txt rules may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores them, so a declared permission can differ from what the fetcher does. ChatGPT-User, Claude-User and Perplexity-User carry each operator's statement.

Agent crawler reachability

Layer and points

Access, up to 2 points. Ora shows it as recommended and verified.

What it reads

Whether the home page is reachable to the same four agents that act for users and search.

Ora's advice

Let the four through the WAF, the bot rules and robots.txt.

Bot-UA markdown serving

Layer and points

Access, up to 1 point. Ora shows it as emerging. A bonus check.

What it reads

Whether the server recognises an AI bot by user agent and answers with markdown even when the bot asks for HTML.

Ora's advice

Optional: detect bot user agents on the server and send a markdown version of the page.

Content without JavaScript

Layer and points

Access, up to 3 points. Ora shows it as required and verified.

What it reads

Whether the home page carries its content in the raw HTML, fetched with JavaScript off.

Ora's advice

Put at least 500 characters of meaningful home page content in the HTML, with a clear h1, and do the same for inner pages.

What reaches your server

The bot-detection and reachability checks send requests from Ora's network. OraBot signs its own requests; the requests that carry other crawlers' user agents are unsigned by design, so a WAF that verifies a claimed GPTBot against OpenAI's addresses will not match them.

Limits

A check result describes what Ora saw from its network on one day. Ora reads public URLs only, says repeated scans of the same site converge within a couple of points (Ora's Deep Scan post), has published no repeat-scan figures on the pages we read, and keeps its fixing product behind a waitlist (Ora's home page). Ora's methodology says only checks it has verified agents rely on count toward the score; the catalog also lists emerging and draft checks, and the layer table above says how many are drafts.

Last reviewed 2026-11-05