--- title: "News: crawler and agent changes | Agent Ready Crawlers" description: "15 dated news items on AI crawlers, agents and bot management, from 2026-08-19 to 2026-11-02, each tied to the Crawler Index rows it changes. Read 2026-11-04." canonical: https://agentreadycrawlers.com/news site: "Agent Ready Crawlers" last_updated: 2026-11-04 date: 2026-11-02 section: "News" --- # News: crawler and agent changes | Agent Ready Crawlers > 15 dated news items on AI crawlers, agents and bot management, from 2026-08-19 to 2026-11-02, each tied to the Crawler Index rows it changes. Read 2026-11-04. Source: https://agentreadycrawlers.com/news Site: Agent Ready Crawlers Last reviewed: 2026-11-04 # News In short 15 dated changes to crawlers, agents and the vendors at the door, from 2026-08-19 to 2026-11-02, newest first. Each item names the rows in the Crawler Index it affects and links its operator or vendor source, read 2026-11-04. The latest, from 2026-11-02, is Signed-Agent-Metadata header proposed for Web Bot Auth. - 2026-11-02[**Signed-Agent-Metadata header proposed for Web Bot Auth**](https://agentreadycrawlers.com/news/2026-11-02-signed-agent-metadata-header-proposed-for-web-bot-auth/) draft-vilgalys-webbotauth-signed-agent-metadata-00 proposes an HTTP header that carries signed metadata about the agent alongside a Web Bot Auth signature. - 2026-10-29[**Vercel splits AI crawlers from AI assistants**](https://agentreadycrawlers.com/news/2026-10-29-vercel-splits-ai-crawlers-from-ai-assistants/) Vercel's AI Bots managed ruleset can now log or block AI crawlers (indexing and training) and AI assistants (user-triggered fetches) independently, on all plans. - 2026-10-29[**Web Bot Auth working-group draft reaches -01**](https://agentreadycrawlers.com/news/2026-10-29-web-bot-auth-working-group-draft-reaches-01/) draft-ietf-webbotauth-httpsig-protocol-01, the adopted working-group draft that replaced the individual Meunier drafts, was posted on 2026-10-29. - 2026-10-27[**OpenAI updates the ChatGPT-User IP list**](https://agentreadycrawlers.com/news/2026-10-27-openai-updates-the-chatgpt-user-ip-list/) openai.com/chatgpt-user.json carried creationTime 2026-10-27 when read on 2026-11-04. - 2026-10-22[**IETF drafts propose a common IP-range format and crawler best practices**](https://agentreadycrawlers.com/news/2026-10-22-ietf-drafts-propose-a-common-ip-range-format-and-crawler-bes/) New webbotauth-related drafts include draft-illyes-webbotauth-jafar-01, a JSON format for publishing IP ranges of automated clients, and draft-illyes-webbotauth-cbcp-01 on crawler best practices, alongside the SAIL source address identity list draft. - 2026-10-21[**Anthropic's Claude Duet starts browsing with a Claude-Agent token**](https://agentreadycrawlers.com/news/2026-10-21-anthropic-s-claude-duet-starts-browsing-with-a-claude-agent/) Claude Duet launched as a waitlist research preview for US Pro and Max subscribers. - 2026-10-20[**Anthropic refreshes its crawler IP list**](https://agentreadycrawlers.com/news/2026-10-20-anthropic-refreshes-its-crawler-ip-list/) claude.com/crawling/bots.json, the list Anthropic links for verifying ClaudeBot, Claude-User and Claude-SearchBot, carried creationTime 2026-10-20 when read, starting with 216.73.216.0/22. - 2026-10-15[**DataDome Q3 report: 25 billion AI agent requests**](https://agentreadycrawlers.com/news/2026-10-15-datadome-q3-report-25-billion-ai-agent-requests/) DataDome counted nearly 25 billion AI agent requests in Q3 2026, up 41.5 percent on the quarter, with Meta-ExternalAgent and Meta-WebIndexer at 49.7 percent. - 2026-10-01[**DataDome joins Experian Agent Trust**](https://agentreadycrawlers.com/news/2026-10-01-datadome-joins-experian-agent-trust/) Experian announced DataDome joined its Agent Trust ecosystem, using verified Know Your Agent signals for continuous intent checks during a session. - 2026-09-17[**Arcjet launches Agent Runtime Security**](https://agentreadycrawlers.com/news/2026-09-17-arcjet-launches-agent-runtime-security/) Arcjet added runtime controls for AI agents that observe activity through OpenTelemetry, enforce policies on tool calls, APIs and databases, and keep an audit trail. - 2026-09-15[**Cloudflare adds Disallow AI Training and the Accountable label**](https://agentreadycrawlers.com/news/2026-09-15-cloudflare-adds-disallow-ai-training-and-the-accountable-lab/) Cloudflare's Training control gains Disallow AI Training, which writes a no-training preference into robots.txt and keeps Accountable mixed-use crawlers (Googlebot, Applebot, Bingbot) in search. - 2026-09-08[**Meta launches Muse, a personal agent that does not identify itself**](https://agentreadycrawlers.com/news/2026-09-08-meta-launches-muse-a-personal-agent-that-does-not-identify-i/) Meta released Muse in the US on iOS, Android, the web and WhatsApp. - 2026-09-02[**HUMAN publishes Agent Trust Levels**](https://agentreadycrawlers.com/news/2026-09-02-human-publishes-agent-trust-levels/) HUMAN's AgenticTrust now grades observed agents High, Medium or Low, with High reserved for agents verifiable by a cryptographic key on each request. - 2026-08-28[**Cloudflare opens BotBase for Operators**](https://agentreadycrawlers.com/news/2026-08-28-cloudflare-opens-botbase-for-operators/) Bot operators get a dashboard home for submissions under Application Security, with status (waiting, accepted, rejected with reasons), editing of existing entries, and a behaviour model to declare how a bot uses content. - 2026-08-19[**Google retires the Google-NotebookLM user agent**](https://agentreadycrawlers.com/news/2026-08-19-google-retires-the-google-notebooklm-user-agent/) Google's user-triggered fetcher list now names Google-GeminiNotebook for URLs users add as sources and marks the former Google-NotebookLM string as supported until August 2026. ## Sources - [IETF webbotauth working group documents](https://datatracker.ietf.org/wg/webbotauth/documents/) read 2026-11-04 - [vercel.com: block ai crawlers and assistants independently with managed rules](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules) read 2026-11-04 - [openai.com: chatgpt user (JSON)](https://openai.com/chatgpt-user.json) read 2026-11-04 - [verisoul.ai: claude duet](https://www.verisoul.ai/agents/claude-duet) read 2026-11-04 - [claude.com: bots (JSON)](https://claude.com/crawling/bots.json) read 2026-11-04 - [helpnetsecurity.com: datadome ai agent traffic report](https://helpnetsecurity.com/2026/10/19/datadome-ai-agent-traffic-report) read 2026-11-04 - [experianplc.com: datadome joins experian agent trust ecosystem to add continuous](https://www.experianplc.com/newsroom/press-releases/2026/datadome-joins-experian-agent-trust--ecosystem-to-add-continuous) read 2026-11-04 - [arcjet.com: introducing agent runtime security observe enforce audit](https://arcjet.com/blog/introducing-agent-runtime-security-observe-enforce-audit) read 2026-11-04 - [Have it both ways: stay discoverable in search while disallowing AI training](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) read 2026-11-04 - [verisoul.ai: meta muse](https://www.verisoul.ai/agents/meta-muse) read 2026-11-04 - [humansecurity.com: evaluating ai agent trust humans agent trust levels](https://www.humansecurity.com/learn/blog/evaluating-ai-agent-trust-humans-agent-trust-levels/) read 2026-11-04 - [BotBase for Operators](https://blog.cloudflare.com/botbase-for-operators/) read 2026-11-04 - [Google user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) read 2026-11-04 Last reviewed 2026-11-04