Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Amazon: 5 rows

In short

Amazon has 5 rows in the Index: Amazonbot, Amzn-SearchBot, Amzn-User, Buy for Me and Rufus. They cover 4 purposes: training, search index, live answer fetch, shopping agent. 3 of them have a robots.txt token and 2 publish nothing a site can verify. The most refused is Amazonbot, disallowed at the root by 15.6 percent of top-10,000 sites.

Amazon: 5 rows.

Operator Token Purpose Respects robots.txt Verification Block rate 2026-11-04 Profile
Amazon Amazonbot Training Yes IP list 15.6% Profile
Amazon Amzn-SearchBot Search index Yes IP list not measured Profile
Amazon Amzn-User Live answer fetch May not apply IP list not measured Profile
Amazon Buy for Me (shopping agent) no token Shopping agent No docs None published no token Profile
Amazon Rufus (shopping assistant) no token Shopping agent No docs None published no token Profile

Vendor profile: Amazon, with its products, standards, our probes and what it does not do.

Verification Ledger entry

Summary
Three IP lists on HTML pages with JSON bodies; nothing for Buy for Me or Rufus.
IP lists
Reverse DNS
publishes nothing
Key directory
publishes nothing
Cloudflare
not checked
Visa and Mastercard
Visa TAP: no public operator-level registration found (checked 2026-11-04). Mastercard Agent Pay: no public operator-level registration found (checked 2026-11-04).

Block rates for Amazon tokens

TokenRefused, top 10,000NamedVia *All crawled
Amazonbot15.6%12.0%3.6%11.7%

Based on 5,938 top-10,000 sites with a parsed robots.txt; crawl of 2026-11-04.

Sources

Last reviewed 2026-11-04