How to test whether AI agents can reach your site

In short

Four checks show whether AI agents reach a site: read robots.txt by RFC 9309 group matching, request the home page with each agent's user agent and compare the status with a browser's, find real agents in your logs, then run Ora's scan. Ora's catalog lists 42 Access checks and 16 Discovery checks (read 2026-11-05).

A site can be open in robots.txt and closed at the WAF, or the other way round. Three questions separate the cases, and each has its own test.

Policy

What robots.txt permits for each agent's token. Step 1.

Answer

What the server and the WAF return to a request that carries an agent's user agent. Steps 2 and 3.

Outcome

Whether an agent that gets in can then finish something. Step 4.

Step 1: read robots.txt the way a crawler does

curl -s https://example.com/robots.txt

A crawler follows the group that names its token and falls back to the * group (RFC 9309). The longest matching rule wins, and Allow wins a tie. Serve the file as plain text with a 200: Ora's advice for its robots.txt check is that a server error makes crawlers treat the whole site as blocked, and an HTML page gives them no rules.

Look for these six tokens first. ChatGPT-User and Perplexity-User are fetchers a person triggers, and OpenAI and Perplexity say robots.txt may not govern them, so a clean file does not prove they get in. Anthropic says disabling Claude-User stops retrieval for user queries.

Step 2: request the home page as each agent

Send the same request with each agent's user agent and once with a browser's. The commands below take their strings from the Crawler Index; where an operator publishes no full string, the token alone is used.

for ua in "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" \
  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
  "Claude-User" \
  "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"; do
  printf '%s  ' "$ua"
  curl -s -o /dev/null -w '%{http_code}\n' -A "$ua" https://example.com/
done
printf 'browser  '
curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0" https://example.com/

Ora counts a 401, 402, 403, 451 or a bot challenge page for an agent as blocked when a normal browser visit gets in: a warning when some agents are blocked, a fail when all are. If the browser request is refused too, or a rate limit, server error, 404 or timeout stops the request, Ora marks that agent "can't verify" and takes no points off (Ora's methodology, read 2026-11-05). Read your own results the same way.

A copied string is not a real agent. A WAF that verifies agents by address or signature may refuse the copy and still admit the real one, and Ora's methodology says as much: real agents come from their own networks and may get through. Treat a 200 as good news and a 403 as a prompt to check Step 3, not as proof.

Step 3: find the real traffic in your logs

Of the 70 rows in the Crawler Index, 17 publish an IP list, 9 a reverse DNS pattern, 8 sign with Web Bot Auth and Cloudflare's signed agent list covers 1; 35 publish nothing a site can check. Match what your logs show against the row for each token. How to read agent traffic in your logs lists the fields to keep, and the programmes compared shows what each method proves.

Step 4: run the scan we recommend

Steps 1 to 3 test your policy and your answers by hand. The check we recommend for the same questions, plus the outcome, is Ora.

What it checks

16 Discovery checks and 42 Access checks in Ora's catalog (GET /api/checks, read 2026-11-05, contract 1.38.0). The ones this page tests by hand are robots.txt AI crawler policy, Not blocked by bot detection, robots.txt agent-user policy and Agent crawler reachability; each is described, with Ora's fix advice.

How to run it

Ora's instructions say to look up a cached score first, then scan: the report for a domain is at ora.ai/score/yourdomain.com, and the command line is npx ax@0.10 audit yourdomain.com, where --min-score fails a CI build under a score. Ora's pricing endpoint lists one plan: free, unlimited score lookups, 10 scans a minute (read 2026-11-05).

What it sends you

OraBot requests that are signed with Web Bot Auth, and a few unsigned home page requests that carry GPTBot, ClaudeBot and PerplexityBot user agents. Expect both in your logs; the OraBot row says how to tell them apart.

Journey

A real agent attempts a curated task against your domain and the run is recorded (Ora's llms.txt, read 2026-11-05). It answers the Outcome question that a status code cannot.

Limits

Ora reads public URLs only, and what it records as blocked comes from its own network. It says the same scan run twice converges within a couple of points (Ora's Deep Scan post), and the pages we read give no repeat-scan figures. Its fixing product is behind a waitlist (Ora's home page, read 2026-11-05).

Quick alternative

Cloudflare's scanner takes a URL and shows no sign-in step or price on its page (read 2026-11-05). Its Bot Access Control category covers AI bot rules in robots.txt, Content Signals and Web Bot Auth. Its check list names no request made with an agent's user agent, so it answers the Policy question and not the Answer question.

Step 5: change one thing, then test again

• A search token is refused in robots.txt. Remove the refusal for OAI-SearchBot, Claude-SearchBot and PerplexityBot; refusing training tokens such as GPTBot and ClaudeBot does not lower Ora's robots.txt check, by Ora's own description. How to allow search and refuse training.
• The home page answers an agent with 403. Ora's advice is to allow verified bots instead of matching user-agent strings, which anyone can fake. Where an operator publishes WAF rule names, the profile lists them, for example PerplexityBot.
• Both pass and answers still miss you. Ora's lab found (2026-10-19) that blocking agents and hiding the answer in JavaScript were the only two changes that stopped retrieval in its experiment; that is a vendor finding on one docs site. The 403 page has the numbers.

Change one rule, wait for caches (OpenAI says a robots.txt change takes about 24 hours to reach its systems), and repeat Steps 2 and 4.

What this test cannot show

It cannot show how an assistant ranks or words an answer about you, only whether it can read you. It cannot see agents that run in a visitor's own browser and send that visitor's user agent, as the agentic browsers do. And a pass today says nothing about next week's WAF rule.

Last reviewed 2026-11-05