The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

News from 2026-10-29

Vercel splits AI crawlers from AI assistants

In short

29 October 2026: Vercel splits AI crawlers from AI assistants. Vercel's AI Bots managed ruleset can now log or block AI crawlers (indexing and training) and AI assistants (user-triggered fetches) independently, on all plans. Vercel verifies known bots by IP, reverse DNS and Web Bot Auth. Vercel's price: basic free; Deep Analysis USD 1 per 1,000 checks on Pro.

Vercel's AI Bots managed ruleset can now log or block AI crawlers (indexing and training) and AI assistants (user-triggered fetches) independently, on all plans. Vercel verifies known bots by IP, reverse DNS and Web Bot Auth.

Primary source vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules, read 2026-11-04.

Where it stands in the Index

Vercel: Vendor. Hosting platform with an invisible CAPTCHA (BotID, Deep Analysis by Kasada), a verified bot directory checked by IP, reverse DNS and Web Bot Auth, and a firewall ruleset that treats AI crawlers and AI assistants separately.

All news.

Sources

vercel.com: block ai crawlers and assistants independently with managed rules read 2026-11-04

Last reviewed 2026-11-04