Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Glossary

Reverse DNS verification

In short

Checking a client IP by looking up its hostname, confirming the operator's domain, then looking the hostname up again to confirm it maps back to the same IP (forward-confirmed reverse DNS). 9 Index rows verify by reverse DNS, among them Googlebot, Google-CloudVertexBot, GoogleOther. Example: Google: host 66.249.66.1 returns crawl-66-249-66-1.googlebot.com, which resolves back to 66.249.66.1.

Definition

Checking a client IP by looking up its hostname, confirming the operator's domain, then looking the hostname up again to confirm it maps back to the same IP (forward-confirmed reverse DNS).

Sourced example

Google: host 66.249.66.1 returns crawl-66-249-66-1.googlebot.com, which resolves back to 66.249.66.1. Source, read 2026-11-04.

How it shows up here

9 Index rows verify by reverse DNS, among them Googlebot, Google-CloudVertexBot, GoogleOther.

Previous: Pay per crawl. Next: RFC 9309. All 41 terms.

Sources

Last reviewed 2026-11-04