News

In short

15 dated changes to crawlers, agents and the vendors at the door, from 2026-08-19 to 2026-11-02, newest first. Each item names the rows in the Crawler Index it affects and links its operator or vendor source, read 2026-11-04. The latest, from 2026-11-02, is Signed-Agent-Metadata header proposed for Web Bot Auth.

• 2026-11-02Signed-Agent-Metadata header proposed for Web Bot Auth
  draft-vilgalys-webbotauth-signed-agent-metadata-00 proposes an HTTP header that carries signed metadata about the agent alongside a Web Bot Auth signature.
• 2026-10-29Vercel splits AI crawlers from AI assistants
  Vercel's AI Bots managed ruleset can now log or block AI crawlers (indexing and training) and AI assistants (user-triggered fetches) independently, on all plans.
• 2026-10-29Web Bot Auth working-group draft reaches -01
  draft-ietf-webbotauth-httpsig-protocol-01, the adopted working-group draft that replaced the individual Meunier drafts, was posted on 2026-10-29.
• 2026-10-27OpenAI updates the ChatGPT-User IP list
  openai.com/chatgpt-user.json carried creationTime 2026-10-27 when read on 2026-11-04.
• 2026-10-22IETF drafts propose a common IP-range format and crawler best practices
  New webbotauth-related drafts include draft-illyes-webbotauth-jafar-01, a JSON format for publishing IP ranges of automated clients, and draft-illyes-webbotauth-cbcp-01 on crawler best practices, alongside the SAIL source address identity list draft.
• 2026-10-21Anthropic's Claude Duet starts browsing with a Claude-Agent token
  Claude Duet launched as a waitlist research preview for US Pro and Max subscribers.
• 2026-10-20Anthropic refreshes its crawler IP list
  claude.com/crawling/bots.json, the list Anthropic links for verifying ClaudeBot, Claude-User and Claude-SearchBot, carried creationTime 2026-10-20 when read, starting with 216.73.216.0/22.
• 2026-10-15DataDome Q3 report: 25 billion AI agent requests
  DataDome counted nearly 25 billion AI agent requests in Q3 2026, up 41.5 percent on the quarter, with Meta-ExternalAgent and Meta-WebIndexer at 49.7 percent.
• 2026-10-01DataDome joins Experian Agent Trust
  Experian announced DataDome joined its Agent Trust ecosystem, using verified Know Your Agent signals for continuous intent checks during a session.
• 2026-09-17Arcjet launches Agent Runtime Security
  Arcjet added runtime controls for AI agents that observe activity through OpenTelemetry, enforce policies on tool calls, APIs and databases, and keep an audit trail.
• 2026-09-15Cloudflare adds Disallow AI Training and the Accountable label
  Cloudflare's Training control gains Disallow AI Training, which writes a no-training preference into robots.txt and keeps Accountable mixed-use crawlers (Googlebot, Applebot, Bingbot) in search.
• 2026-09-08Meta launches Muse, a personal agent that does not identify itself
  Meta released Muse in the US on iOS, Android, the web and WhatsApp.
• 2026-09-02HUMAN publishes Agent Trust Levels
  HUMAN's AgenticTrust now grades observed agents High, Medium or Low, with High reserved for agents verifiable by a cryptographic key on each request.
• 2026-08-28Cloudflare opens BotBase for Operators
  Bot operators get a dashboard home for submissions under Application Security, with status (waiting, accepted, rejected with reasons), editing of existing entries, and a behaviour model to declare how a bot uses content.
• 2026-08-19Google retires the Google-NotebookLM user agent
  Google's user-triggered fetcher list now names Google-GeminiNotebook for URLs users add as sources and marks the former Google-NotebookLM string as supported until August 2026.

Last reviewed 2026-11-04