Questions and answers

What is ClaudeBot?

ClaudeBot is Anthropic's training crawler. Restricting it in robots.txt signals that future content should be excluded from Anthropic training datasets. Anthropic runs two other bots: Claude-User for user-requested fetches and Claude-SearchBot for search indexing. Source addresses are listed at claude.com/crawling/bots.json. Source, read 2026-11-04.

How do I block GPTBot without leaving ChatGPT search?

Add "User-agent: GPTBot" then "Disallow: /" and keep OAI-SearchBot allowed. OpenAI treats the two settings as independent, and says robots.txt changes take about 24 hours to reach its systems. ChatGPT-User is a separate user-triggered fetcher that may not apply robots.txt. Source, read 2026-11-04.

Does ChatGPT-User obey robots.txt?

Not necessarily. OpenAI says that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply, and it is not used to decide search inclusion. To refuse it you need a WAF rule that matches its user agent and the ranges in chatgpt-user.json. Source, read 2026-11-04.

Which AI fetchers say they ignore robots.txt for user requests?

OpenAI (ChatGPT-User, rules may not apply), Perplexity (Perplexity-User generally ignores robots.txt), Google (user-triggered fetchers generally ignore it), Amazon (Amzn-User may not follow all directives) and Meta (Meta-ExternalFetcher may bypass it). Anthropic is the exception: it says Claude-User lets site owners control user-initiated access. Source, read 2026-11-04.

How can I tell a real Googlebot from a fake one?

Run a reverse DNS lookup on the IP; it must end in googlebot.com, google.com or googleusercontent.com. Then run a forward lookup on that hostname and confirm it returns the same IP. For volume, match against Google's published JSON lists for common crawlers, special crawlers, user-triggered fetchers and user-triggered agents. Source, read 2026-11-04.

What is Web Bot Auth?

Web Bot Auth is an IETF protocol where a bot signs each request with HTTP Message Signatures (RFC 9421) and names its key directory in a Signature-Agent header. The site fetches the public keys and verifies the signature. The working group draft draft-ietf-webbotauth-httpsig-protocol reached revision -01 on 2026-10-29. Source, read 2026-11-04.

Which operators sign their agent traffic today?

Read on 2026-11-04: OpenAI's Cloud browser (keys at chatgpt.com), Google-Agent for a subset of requests (agent.bot.goog, experimental), Browserbase (keys at browserbase.com), Cloudflare Browser Rendering, AWS AgentCore Browser (preview, opt-in) and Apify crawlers when signHttpRequests is on. Anthropic, Perplexity, Meta and Microsoft publish no key directory for their agents. Source, read 2026-11-04.

Can I block Claude in Chrome or Comet?

Not by identity. Both run inside the user's own browser and send that browser's user agent, cookies and IP, and neither publishes a token or signature for site owners. Detection vendors flag them by behaviour. Rules aimed at them will also hit the human using the browser. Source, read 2026-11-04.

What does Google-Agent do?

Google-Agent is the user-triggered agent Google uses for agents hosted on its infrastructure that navigate the web and act on user request. It comes from the ranges in user-triggered-agents.json, and a subset of its requests is signed with Web Bot Auth as https://agent.bot.goog. Google calls the signing experimental. Source, read 2026-11-04.

Does blocking Google-Extended hurt my Google rankings?

Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. It controls whether crawled content trains Gemini models and grounds Gemini Apps and Vertex AI answers. Source, read 2026-11-04.

How do I stop Apple training on my site but stay in Siri and Spotlight?

Disallow Applebot-Extended in robots.txt and keep Applebot allowed. Applebot-Extended crawls nothing; it only governs how Applebot data is used. Apple also reads nosnippet to keep content out of broad world knowledge answers. Source, read 2026-11-04.

How do I opt out of Bing training?

There is no robots.txt token for it yet. Use the NOARCHIVE meta tag, which Microsoft says does not affect ranking. Cloudflare reports Microsoft is building robots.txt support for a no-training preference, targeted for early 2027. Source, read 2026-11-04.

What changed in Cloudflare's AI crawler settings on 15 September 2026?

Cloudflare added Disallow AI Training, made Block and Block on pages with ads apply to mixed-use crawlers such as Googlebot, Bingbot and Applebot, deprecated the Block AI Bots toggle in favour of Search, Training and Agent controls, and replaced Managed Robots.txt with Bot Preference Sync. Source, read 2026-11-04.

Is Cloudflare pay per crawl available to everyone?

No. Cloudflare's docs (updated 2026-07-28) say pay per crawl is in closed beta and site owners must apply or go through their account executive. Crawlers get HTTP 402 with a crawler-price header and Cloudflare acts as Merchant of Record. Source, read 2026-11-04.

Can AWS WAF charge AI bots?

Yes, since 2026-06-15. AWS WAF AI traffic monetization returns an x402 HTTP 402 with prices and terms, settles through Coinbase in stablecoins, and works only on web ACLs attached to CloudFront. AWS charges nothing extra for it beyond WAF fees. Source, read 2026-11-04.

Does Meta publish IP ranges for its AI crawlers?

Not on its crawler page read on 2026-11-04. Meta lists the user agent strings for FacebookExternalHit, Meta-WebIndexer, Meta-ExternalAds, Meta-ExternalAgent and Meta-ExternalFetcher and says IP allowlisting is more secure, but links no IP file or reverse DNS pattern. Source, read 2026-11-04.

What is the difference between Meta-ExternalAgent and Meta-ExternalFetcher?

Meta-ExternalAgent crawls for uses such as training foundation models or indexing content. Meta-ExternalFetcher fetches individual links at a user's request, including helping AI complete tasks, and Meta says it may bypass robots.txt. Source, read 2026-11-04.

Does Bytespider respect robots.txt?

ByteDance publishes no documentation for Bytespider, so there is no operator statement to check against (checked 2026-11-04). Treat any claim about its behaviour as third-party observation, and enforce at the WAF if you need it gone. Source, read 2026-11-04.

How do I verify CCBot?

Common Crawl runs CCBot from dedicated ranges with reverse DNS under crawl.commoncrawl.org (IPv4 only) and publishes the ranges at index.commoncrawl.org/ccbot.json. It warns that other crawlers falsely identify as CCBot. Source, read 2026-11-04.

Which crawlers support Crawl-delay?

Anthropic supports it ("User-agent: ClaudeBot" then "Crawl-delay: 1"); ImagesiftBot supports it as minimum seconds between request starts; Amazon says Amazonbot, Amzn-SearchBot and Amzn-User do not. RFC 9309 does not define Crawl-delay. Source, read 2026-11-04.

How long do robots.txt changes take to apply?

OpenAI and Perplexity say about 24 hours; Amazon says about 24 hours and may use a cached copy up to 30 days old; Meta says up to 24 hours; DuckDuckGo says 72 hours for DuckAssistBot. Source, read 2026-11-04.

What does a Cloudflare signed agent mean for my rules?

Signed agents are agents directed by end users whose requests are signed with Web Bot Auth and validated by Cloudflare. Since 2026-07-01 they sit under the verified bot umbrella, so you can allow or skip them with rules on verified status or a specific detection ID. Source, read 2026-11-04.

How do I allow ChatGPT's Cloud browser on Cloudflare?

OpenAI says create a WAF custom rule on Bot Detection ID equals 129220581 (bot tag chatgpt-agent) and set it to allow or skip. Cloudflare verifies the signature itself. Source, read 2026-11-04.

Does Vercel block AI crawlers?

Vercel's AI Bots managed ruleset can log or block AI crawlers (indexing and training) and AI assistants (user-triggered fetches) separately, on all plans, since 2026-10-29. Vercel verifies known bots by IP, reverse DNS and Web Bot Auth. Source, read 2026-11-04.

What is the Verification Ledger on this site?

A table of which operators publish verification material (IP JSON, reverse DNS pattern, Web Bot Auth key directory, Cloudflare verified status, Visa TAP or Mastercard Agent Pay registration), with the exact URL and the date it was read, and which publish nothing. Source, read 2026-11-04.

What is Visa Trusted Agent Protocol?

A Visa specification for AI agents to present a cryptographic signature with timestamps to merchants, so a merchant can tell a credentialed agent with user permission from an anonymous bot. Visa publishes reference code on GitHub; Cloudflare says it is built on Web Bot Auth. Source, read 2026-11-04.

What does a robots.txt Content-Signal line do?

It states preferences for search, ai-input and ai-train under the Content Signals Policy, for example "Content-Signal: search=yes, ai-train=no". It identifies nobody and stops nobody; Cloudflare says enforcement needs a network that identifies the crawler. Source, read 2026-11-04.

Does returning 403 to AI agents hurt how assistants describe my business?

Ora's lab found (2026-10-19) that a 403 to agent user agents left both test clients with 0 of 5 answers retrieved, one of only two changes that broke retrieval in its experiment. That is a vendor finding on one docs site; test your own pages. Source, read 2026-11-04.

Which AI crawler sends the most traffic?

It depends on whose network you ask. DataDome's Q3 2026 report put Meta-ExternalAgent and Meta-WebIndexer at 49.7 percent of nearly 25 billion AI agent requests; HUMAN's August 2026 data put Comet at 40.7 percent of agentic browser traffic. Both are vendor figures from their own customers. Source, read 2026-11-04.

Where do the block rates on this site come from?

From the network's shared crawl of 2026-11-04: a plain fetch of robots.txt for the Tranco top 10,000 plus leaderboard and vendor domains, parsed per token. The crawl stores whether each token is disallowed at the root and whether it is named explicitly. Source, read 2026-11-04.

How do I test whether AI agents can reach my site?

Read your robots.txt for the tokens that matter, request the home page once with each agent's user agent and once as a browser, and compare the status codes. Then look for the real agents in your logs and run a scan such as Ora's. A copied user agent is not a real agent, so a 403 on the copy is a prompt to check the logs before changing a rule. Source, read 2026-11-05.

Does Ora's scanner sign its requests?

Ora says every request that carries the OraBot user agent is signed with HTTP Message Signatures using an Ed25519 key published in its key directory. During a scan it also sends a few unsigned home page requests that carry other crawlers' user agents, such as GPTBot and ClaudeBot, to see how a site answers them. Source, read 2026-11-05.

Which AI bots should a publisher allow?

A publisher can refuse the training tokens and keep AI search and user-triggered fetches open. OpenAI treats OAI-SearchBot and GPTBot as independent settings, and says sites opted out of OAI-SearchBot do not appear in ChatGPT search answers. The site-type guide writes a robots.txt for five kinds of site from the Crawler Index. Source, read 2026-11-04.