Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

News from 2026-10-22

IETF drafts propose a common IP-range format and crawler best practices

In short

22 October 2026: IETF drafts propose a common IP-range format and crawler best practices. New webbotauth-related drafts include draft-illyes-webbotauth-jafar-01, a JSON format for publishing IP ranges of automated clients, and draft-illyes-webbotauth-cbcp-01 on crawler best practices, alongside the SAIL source address identity list draft. No Index row or vendor profile is tied to it; read the source before changing any rule.

New webbotauth-related drafts include draft-illyes-webbotauth-jafar-01, a JSON format for publishing IP ranges of automated clients, and draft-illyes-webbotauth-cbcp-01 on crawler best practices, alongside the SAIL source address identity list draft.

Primary source datatracker.ietf.org/wg/webbotauth/documents/, read 2026-11-04.

All news.

Sources

Last reviewed 2026-11-04