Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Agent verification provider

Mastercard

In short

Mastercard tells merchants not to block consumer-owned agents wholesale, comparing that with blocking mobile browsers in 2010, and to add friction at checkout instead. In the simplest integration the merchant sees ordinary card fields filled with tokenized credentials and can check verification data from request headers. The site refused our crawl, and the playbook's example key directory does not resolve.

Type
Verification provider
Homepage
developer.mastercard.com
In one line
Card network whose Agent Pay registers AI agents and has them sign requests with Web Bot Auth under the tag Agent-pay-auth, so a merchant or its CDN can check the agent's key in a network directory before letting it buy.
Last read
2026-11-05

What Mastercard does for this layer

From the vendor's own pages

Mastercard's Developer Playbook on Preparing for Agentic Commerce names three network frameworks a merchant will meet: Mastercard Agent Pay, Visa Intelligent Commerce with the Trusted Agent Protocol, and the American Express ACE Developer Kit with Agent Purchase Protection. (developer.mastercard.com) Mastercard's playbook lists Agent-pay-auth as the Signature-Input tag for Mastercard Agent Pay, next to Visa's agent-browser-auth for browsing and agent-payer-auth for checkout. (developer.mastercard.com) The example signed request in Mastercard's playbook carries the header Signature-Agent: "https://agentpay-key-directory.mastercard.com/" with an Ed25519 keyid, created and expires timestamps and a nonce. (developer.mastercard.com) Mastercard's playbook says agents that implement Web Bot Auth on their own, such as ChatGPT, use the generic web-bot-auth tag, and tells merchants to accept both the scheme tags and the generic one. (developer.mastercard.com)

In Mastercard's registration model an agent creates a key pair at onboarding and registers its public key in a payment network key directory; the merchant's edge provider or backend fetches that key to validate each signature. (developer.mastercard.com) Mastercard's playbook names Cloudflare, Akamai and Fastly as edge providers that support Web Bot Auth verification at the network edge, as the alternative to verifying in application code. (developer.mastercard.com) At Mastercard's Level 1, an agent using Agent Pay types tokenized payment credentials into the merchant's normal checkout fields while hash verification data arrives in HTTP headers that the merchant backend may validate. (developer.mastercard.com) Mastercard's playbook calls indiscriminate blocking of consumer-owned agents such as OpenClaw, Agent Zero and Hermes Agent "the equivalent of blocking mobile browsers in 2010" and recommends open browsing with progressive friction at checkout. (developer.mastercard.com)

Mastercard also describes merchant registration: the acquirer or PSP submits the merchant to the network's agentic commerce registry, and once verified the merchant appears in a trusted merchant directory that certified agents query at checkout. (developer.mastercard.com)

Reported by a third party

Cloudflare wrote on 24 Oct 2025 that, to start, Visa and Mastercard host their own key directories for Visa-registered and Mastercard-registered agents. (blog.cloudflare.com, dated 2025-10-24)

What our crawl saw

On 2026-11-05 a GET of https://www.mastercard.com/robots.txt with the crawl user agent returned HTTP 403 with an Akamai Access Denied page, so Mastercard's Agent Pay product pages on that host were not read for this profile. (mastercard.com) On 2026-11-05 the host agentpay-key-directory.mastercard.com, used in the playbook's example Signature-Agent header, did not resolve in DNS, so no Mastercard agent key directory could be fetched. (developer.mastercard.com)

Products and plans

ProductWhat it doesPrice modelFromAs of
Mastercard Agent Pay Agent registration and Web Bot Auth signing with the Agent-pay-auth tag, with tokenized credentials entered at checkout. No price recorded
Developer Playbook on Preparing for Agentic Commerce Public merchant guide on developer.mastercard.com covering three readiness levels, Web Bot Auth verification and merchant registration. No price recorded

Standards

StandardStatusEvidence
Web Bot Auth Supports The playbook builds agent identification on Web Bot Auth and RFC 9421 and accepts the generic web-bot-auth tag. (developer.mastercard.com)
Visa Trusted Agent Protocol Partial The playbook documents Visa's tags so merchants can verify both schemes; Mastercard does not run TAP. (developer.mastercard.com)
Mastercard Agent Pay Supports Mastercard defines the Agent Pay tag Agent-pay-auth and the agent key directory model in its merchant playbook. (developer.mastercard.com)

Our checks

CheckResultMethodDate
Homepage fetch HTTP 200 GET with the crawl user agent after a robots.txt check 2026-11-05
Crawl access to www.mastercard.com HTTP 403, Akamai Access Denied, robots.txt included GET https://www.mastercard.com/robots.txt with the crawl user agent 2026-11-05
Agent Pay key directory host agentpay-key-directory.mastercard.com does not resolve DNS lookup and GET of /.well-known/http-message-signatures-directory 2026-11-05

What Mastercard does not do

  • Mastercard does not verify signatures for the merchant; the merchant's CDN or backend does the check against the network's key directory.
  • Mastercard publishes no public list of registered agents or a reachable key directory URL that a site can test against (checked 2026-11-05).
  • Agent Pay covers agents with payment intent; it gives no signal for training or search crawlers.

Reading notes

www.mastercard.com and businesswire.com refused our crawl user agent (a 403, and a robots.txt disallow), so press releases were not read; the facts come from developer.mastercard.com, which allows the crawl and serves markdown.

Sources

Last reviewed 2026-11-05