Guides

In short

14 guides for the people who own robots.txt and the WAF: reading agent traffic in logs, allowing search while refusing training, verifying Web Bot Auth, letting signed commerce agents through, testing what a 403 does and testing whether agents can reach a site. Each ends with its sources and dates.

• How to read agent traffic in your logs
  A user-agent string is a claim. A log line becomes evidence when the token, the source address and, for signed agents, the Signature-Agent header agree.
• How to allow search and answer bots while refusing training
  Operators have split search from training into separate tokens. Most of the work is naming the training tokens and leaving the search and fetch tokens alone.
• How to verify Web Bot Auth signatures
  Web Bot Auth lets an operator prove a request is its own with a signature. Verification is five steps and the first is reading one header.
• How to let Visa TAP and Mastercard Agent Pay agents through
  Visa and Mastercard sign their agents' requests, so the question for a merchant is whether the CDN or bot manager already checks those signatures.
• What pay per crawl pays and who is paying
  Three ways to charge a crawler exist today. None publishes payout totals, so what they pay publishers is not yet public.
• How to test what a 403 does to an AI answer about you
  A 403 to an agent is invisible in your analytics and decisive for the answer. The test takes an afternoon and needs no tool.
• The difference between a crawler and a browsing agent
  A crawler fetches on a schedule under its own name. A browsing agent acts for one person and often borrows that person's browser.
• How robots.txt tokens differ from user-agent strings
  The word after User-agent in robots.txt is not what you see in the logs, and some tokens never appear there at all.
• How Cloudflare's verified bot list works
  Cloudflare's verified list is the closest thing to a shared register of good bots. How a bot gets on it decides what your rules can trust.
• What Content-Signals changes
  Content-Signal lines add a vocabulary for what a site allows after it has been crawled. They state a preference and enforce nothing.

Added on 2026-11-05

• How to test whether AI agents can reach your site
  Read robots.txt, request the home page as each agent, check your logs, then run the scan we recommend.
• Which AI bots to allow, by type of site
  Five kinds of site, the purposes to allow or refuse for each, and a robots.txt generated from the Index for every one.
• 403 to AI agents: what the crawl and Ora show
  What a 403 to an agent user agent does, the share of home pages that answer one, and how to find yours.
• The Ora checks that touch bot access
  Seven checks from Ora's catalog that read robots.txt, user agents and the home page, with Ora's own fix advice.

Last reviewed 2026-11-05