The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Glossary

Bot management

In short

Products that classify automated traffic and apply allow, challenge, rate-limit or block decisions, increasingly with separate handling for verified AI crawlers and agents. 1.3 percent of home pages carried a header or cookie of a bot product (DataDome, HUMAN, Imperva, Kasada or Arcjet) and 6.1 percent a Cloudflare challenge header. Example: Fastly exposes VERIFIED-BOT.AI-CRAWLER and SUSPECTED-BOT.AI-FETCHER signals.

Definition

Products that classify automated traffic and apply allow, challenge, rate-limit or block decisions, increasingly with separate handling for verified AI crawlers and agents.

Sourced example

Fastly exposes VERIFIED-BOT.AI-CRAWLER and SUSPECTED-BOT.AI-FETCHER signals. Source, read 2026-11-04.

How it shows up here

1.3 percent of home pages carried a header or cookie of a bot product (DataDome, HUMAN, Imperva, Kasada or Arcjet) and 6.1 percent a Cloudflare challenge header.

Previous: Applebot-Extended. Next: Bot paywall. All 41 terms.

Sources

fastly.com: how to control and monetize ai bot traffic using fastly and tollbit read 2026-11-04

Last reviewed 2026-11-04