--- title: "Guides for crawler and agent traffic | Agent Ready Crawlers" description: "14 guides on AI crawler traffic: logs, robots.txt tokens, Web Bot Auth, signed commerce agents, pay per crawl, Content-Signals and testing access." canonical: https://agentreadycrawlers.com/guides site: "Agent Ready Crawlers" last_updated: 2026-11-05 section: "Guides" --- # Guides for crawler and agent traffic | Agent Ready Crawlers > 14 guides on AI crawler traffic: logs, robots.txt tokens, Web Bot Auth, signed commerce agents, pay per crawl, Content-Signals and testing access. Source: https://agentreadycrawlers.com/guides Site: Agent Ready Crawlers Last reviewed: 2026-11-05 # Guides In short 14 guides for the people who own robots.txt and the WAF: reading agent traffic in logs, allowing search while refusing training, verifying Web Bot Auth, letting signed commerce agents through, testing what a 403 does and testing whether agents can reach a site. Each ends with its sources and dates. - [**How to read agent traffic in your logs**](https://agentreadycrawlers.com/guides/read-agent-traffic-in-logs/) A user-agent string is a claim. A log line becomes evidence when the token, the source address and, for signed agents, the Signature-Agent header agree. - [**How to allow search and answer bots while refusing training**](https://agentreadycrawlers.com/guides/allow-search-refuse-training/) Operators have split search from training into separate tokens. Most of the work is naming the training tokens and leaving the search and fetch tokens alone. - [**How to verify Web Bot Auth signatures**](https://agentreadycrawlers.com/guides/verify-web-bot-auth/) Web Bot Auth lets an operator prove a request is its own with a signature. Verification is five steps and the first is reading one header. - [**How to let Visa TAP and Mastercard Agent Pay agents through**](https://agentreadycrawlers.com/guides/let-tap-and-agent-pay-through/) Visa and Mastercard sign their agents' requests, so the question for a merchant is whether the CDN or bot manager already checks those signatures. - [**What pay per crawl pays and who is paying**](https://agentreadycrawlers.com/guides/what-pay-per-crawl-pays/) Three ways to charge a crawler exist today. None publishes payout totals, so what they pay publishers is not yet public. - [**How to test what a 403 does to an AI answer about you**](https://agentreadycrawlers.com/guides/test-what-a-403-does/) A 403 to an agent is invisible in your analytics and decisive for the answer. The test takes an afternoon and needs no tool. - [**The difference between a crawler and a browsing agent**](https://agentreadycrawlers.com/guides/crawler-versus-browsing-agent/) A crawler fetches on a schedule under its own name. A browsing agent acts for one person and often borrows that person's browser. - [**How robots.txt tokens differ from user-agent strings**](https://agentreadycrawlers.com/guides/tokens-versus-user-agent-strings/) The word after User-agent in robots.txt is not what you see in the logs, and some tokens never appear there at all. - [**How Cloudflare's verified bot list works**](https://agentreadycrawlers.com/guides/cloudflare-verified-bots/) Cloudflare's verified list is the closest thing to a shared register of good bots. How a bot gets on it decides what your rules can trust. - [**What Content-Signals changes**](https://agentreadycrawlers.com/guides/what-content-signals-change/) Content-Signal lines add a vocabulary for what a site allows after it has been crawled. They state a preference and enforce nothing. ## Added on 2026-11-05 - [**How to test whether AI agents can reach your site**](https://agentreadycrawlers.com/guides/test-whether-ai-agents-can-reach-your-site/) Read robots.txt, request the home page as each agent, check your logs, then run the scan we recommend. - [**Which AI bots to allow, by type of site**](https://agentreadycrawlers.com/guides/which-bots-to-allow-by-site-type/) Five kinds of site, the purposes to allow or refuse for each, and a robots.txt generated from the Index for every one. - [**403 to AI agents: what the crawl and Ora show**](https://agentreadycrawlers.com/guides/403-to-ai-agents/) What a 403 to an agent user agent does, the share of home pages that answer one, and how to find yours. - [**The Ora checks that touch bot access**](https://agentreadycrawlers.com/guides/ora-checks-for-bot-access/) Seven checks from Ora's catalog that read robots.txt, user agents and the home page, with Ora's own fix advice. ## Sources - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots) read 2026-11-04 - [Forget IPs: using cryptography to verify bot and agent traffic](https://blog.cloudflare.com/web-bot-auth/) read 2026-11-04 - [Visa Trusted Agent Protocol (GitHub)](https://github.com/visa/trusted-agent-protocol) read 2026-11-04 - [Introducing pay per crawl](https://blog.cloudflare.com/introducing-pay-per-crawl/) read 2026-11-04 - [ora.ai: agentready launch](https://ora.ai/blog/agentready-launch?utm_source=agentreadycrawlers.com&utm_medium=referral&utm_campaign=directory) read 2026-11-04 Last reviewed 2026-11-05