Glossary
In short
41 terms for reading robots.txt, crawler logs and bot-management settings, from Applebot-Extended to Web Bot Auth. Each has a sourced example, read 2026-11-04, and a note on how it shows up in the Crawler Index or the crawl behind the Block Rate Report.
- Accountable designation
- Cloudflare's label for crawler operators that offer a training opt-out, an AI-summary opt-out, URL-level visibility, and an assurance that opting out of training does not affect search.
- Agentic browser
- A consumer browser with a built-in agent that can act on pages, such as Comet, Gemini in Chrome or Browse with Copilot.
- ai-input
- The Content Signals value for using content as live input to an AI answer, such as grounding or retrieval.
- AIPREF training signal
- An IETF AI Preferences vocabulary for usage preferences.
- Applebot-Extended
- Apple's control token for whether Applebot data trains Apple foundation models.
- Bot management
- Products that classify automated traffic and apply allow, challenge, rate-limit or block decisions, increasingly with separate handling for verified AI crawlers and agents.
- Bot paywall
- A redirect target where AI bots are told the price for content and must present a valid token to get it.
- Bot Preference Sync
- The Cloudflare feature that writes a zone's crawler preferences into the robots.txt it serves, replacing Managed Robots.txt from 2026-09-15.
- Bot score
- A numeric likelihood that a request is automated, used in rules.
- BotBase
- Cloudflare's directory of known bots and agents, with an operator portal (launched 2026-08-28) for submissions, status and edits.
- Browsing agent
- Software that drives a browser to complete a task for a person: clicking, typing and checking out.
- Content-Signal
- A robots.txt line from the Content Signals Policy that states yes or no for three uses: search, ai-input and ai-train.
- Crawl-delay
- A non-standard robots.txt line asking a crawler to wait a number of seconds between requests.
- crawler-price header
- The response header Cloudflare pay per crawl sends with a 402 to state the per-request price.
- Disallow AI Training
- A Cloudflare Training-control setting launched 2026-09-15 that publishes a no-training preference in robots.txt, keeps Accountable mixed-use crawlers for search and blocks other training crawlers.
- Google-Extended
- Google's control token for whether crawled content may train Gemini models and ground Gemini Apps and Vertex AI answers.
- HTTP 402 Payment Required
- An HTTP status code reserved since the 1990s and now used by pay-per-crawl and x402 to tell a client that access needs payment.
- HTTP Message Signatures
- RFC 9421: a standard for signing selected components of an HTTP message, carried in Signature and Signature-Input headers.
- IP range JSON
- A JSON file of CIDR prefixes an operator publishes for one crawler, usually with a creationTime field.
- Key directory
- A JSON document at /.well-known/http-message-signatures-directory listing a signer's public keys for Web Bot Auth, served with the media type application/http-message-signatures-directory+json.
- Know Your Agent (KYA)
- An identity scheme in which an issuer vouches for an agent, its platform and the user it represents, carried as a signed token.
- KYAPay
- The token form that combines KYA identity with payment intent and committed funds, so a seller can verify and charge in one step.
- Mastercard Agent Pay
- Mastercard's programme for registered agents to transact, which signs agent traffic with Web Bot Auth so merchants and CDNs can verify it.
- Mixed-use crawler
- A single crawler whose content serves both search and AI training, so blocking it to refuse training also removes the site from search.
- noarchive
- A robots meta directive.
- nosnippet
- A robots meta directive that limits snippets.
- Pay per crawl
- Cloudflare's closed-beta feature that lets a zone charge AI crawlers per request using HTTP 402, with Cloudflare as Merchant of Record.
- Reverse DNS verification
- Checking a client IP by looking up its hostname, confirming the operator's domain, then looking the hostname up again to confirm it maps back to the same IP (forward-confirmed reverse DNS).
- RFC 9309
- The IETF Robots Exclusion Protocol standard, published September 2022, that defines robots.txt groups, Allow and Disallow rules, longest-match precedence and case-insensitive product-token matching.
- robots.txt
- A plain-text file at the root of a host that tells crawlers which paths they may fetch, grouped by user-agent token.
- Robots.txt token
- The name a crawler answers to in a User-agent line.
- Search-index crawler
- A crawler that builds an index used to surface and cite pages in search or AI answers, documented by its operator as not used for training.
- Signature-Agent header
- The request header in Web Bot Auth that names the URL of the signer's key directory, telling the site where to fetch public keys.
- Signed agent
- Cloudflare's class for agents directed by end users whose requests are signed with Web Bot Auth and who follow its signed agent policy.
- Spoofed crawler
- A request that copies a known crawler's user agent to get treatment meant for the real bot.
- Training crawler
- A crawler whose fetched content may be used to train AI models.
- User-agent string
- The User-Agent HTTP header a client sends.
- User-triggered fetcher
- A fetch made because a person asked an assistant something, such as reading a linked page.
- Verified bot
- A bot that a security provider has confirmed belongs to its stated operator, by IP validation or signature, and lists in a directory site owners can write rules against.
- Visa Trusted Agent Protocol
- Visa's specification for agents to present a signature with timestamps that merchants verify to tell a credentialed agent from an anonymous bot.
- Web Bot Auth
- An IETF protocol in which bots and agents sign HTTP requests with HTTP Message Signatures and publish their public keys, so sites can verify the operator without trusting IPs or user agents.
Sources
Last reviewed 2026-11-04