Crawler operator
Amazon
In short
Amazon publishes three crawler identities with separate robots.txt settings: Amazonbot may feed model training, while Amzn-SearchBot and Amzn-User do not. None of them supports crawl-delay, and robots.txt copies can be 30 days old. Buy for Me, in beta since April 2025, buys from brand sites for shoppers but has no documented identifier.
- Type
- Crawler operator
- Homepage
- amazon.com
- In one line
- Amazon runs Amazonbot, whose content may train Amazon AI models, plus Amzn-SearchBot for Alexa search and Amzn-User for live answers, each with an IP list, and buys from brand sites for shoppers through its Buy for Me agent.
- Last read
- 2026-11-05
What Amazon does for this layer
From the vendor's own pages
Amazon documents three user agents, Amazonbot, Amzn-SearchBot and Amzn-User, says each setting is independent, and says changes may take about 24 hours to reach its systems. (developer.amazon.com) Amazon says Amazonbot content may be used to train Amazon AI models, while Amzn-SearchBot and Amzn-User do not crawl content for generative AI model training. (developer.amazon.com) Amazon says that when robots.txt does not mention Amzn-SearchBot but allows other search bots, Amzn-SearchBot follows the directives given to those bots. (developer.amazon.com) Amazon's crawlers fetch robots.txt per host or use a cached copy up to 30 days old, treat an unfetchable file as absent, and do not support crawl-delay. (developer.amazon.com)
Amazon's crawlers honour rel=nofollow and the robots meta values noarchive, noindex and none, and Amazon reads noarchive as do not use the page for model training. (developer.amazon.com) Amazon launched Buy for Me in beta on 2025-04-03 to buy items from other brands' websites on a customer's behalf, and says the feature runs on Amazon Bedrock with Anthropic's Claude models. (aboutamazon.com, dated 2025-04-03) Amazon's Shop Direct Merchant Workspace, announced 2026-11-04, lets eligible merchants register their stores for Shop Direct, whose products may be bought through Buy for Me; Amazon says there are no fees and merchants can opt out at any time. (aboutamazon.com, dated 2026-11-04)
Claims the vendor makes about itself
Amazon's crawler page says sites that allow Amazonbot may qualify for Amazon Content Partners, which offers a "+1% affiliate commission boost", hosting credits and AI traffic management tools. (developer.amazon.com)
What our crawl saw
Amazon's Amzn-User range page embedded a JSON object with 1,023 IPv4 prefixes and a creationTime of 2025-11-04 when fetched on 2026-11-05. (developer.amazon.com, dated 2025-11-04) www.amazon.com answered HTTP 404 at the Web Bot Auth key directory path on 2026-11-05. (amazon.com)
Products and plans
| Product | What it does | Price model | From | As of |
|---|---|---|---|---|
| Amazonbot | Crawler that improves Amazon products and may train Amazon AI models. | |||
| Amzn-SearchBot | Crawler for search experiences such as Alexa; not used for generative model training. | |||
| Amzn-User | Fetcher for live answers to Alexa queries; may not follow all robots.txt directives. | |||
| Buy for Me | Agent in the Amazon Shopping app that completes purchases on brand websites for the customer. | |||
Standards
| Standard | Status | Evidence |
|---|---|---|
| RFC 9309 (robots.txt) | Partial | Automated crawling honours the Robots Exclusion Protocol; Amzn-User may not follow all directives when a user starts the action. (developer.amazon.com) |
| Web Bot Auth | No statement | Not on the crawler page; key directory path 404 on www.amazon.com (observed 2026-11-05). |
| Published IP list | Supports | Three range pages, one per user agent, each embedding a JSON object. (developer.amazon.com) |
| Reverse DNS check | No statement | No reverse DNS pattern on the crawler page (read 2026-11-05). |
Our checks
| Check | Result | Method | Date |
|---|---|---|---|
| Homepage fetch | HTTP 202 | GET with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | 404 | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 |
| Amazonbot range page | 1,292 IPv4 prefixes, creationTime 2026-09-08 | GET and count prefixes in the embedded JSON | 2026-11-05 |
| Amzn-SearchBot range page | 816 IPv4 prefixes, creationTime 2026-09-08 | GET and count prefixes in the embedded JSON | 2026-11-05 |
| Amzn-User range page | 1,023 IPv4 prefixes, creationTime 2025-11-04 | GET and count prefixes in the embedded JSON | 2026-11-05 |
What Amazon does not do
- Amazon's crawler page documents no identifier for Buy for Me or Rufus traffic (read 2026-11-05).
- Amazon names no robots.txt token for Buy for Me; the opt-out it describes is for merchants enrolled in Shop Direct.
- Amazon's crawlers ignore crawl-delay, so rate limits need server-side controls.
Reading notes
Amazon's cloud products, AWS WAF and AgentCore Browser, are profiled under AWS WAF Bot Control.
Related pages on this site
Sources
- amazon.com
- Amazon Web Search crawlers (Amazonbot)
- Amazonbot Live Crawl IP addresses (Amzn-User)
- Amazonbot IP addresses
- Amazon's new Buy for Me feature helps customers find and buy products from other brands' sites
- Amazon launches Shop Direct Merchant Workspace for merchants
- www.amazon.com Web Bot Auth key directory path (GET probe)
- Amazon Searchbot IP addresses
Last reviewed 2026-11-05