Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

How to let Visa TAP and Mastercard Agent Pay agents through

In short

Both programmes sign agent requests with Web Bot Auth tags; check whether your CDN or bot manager verifies them before writing custom rules. Visa's Trusted Agent Protocol gives agents a cryptographic signature with timestamps that merchants verify to tell a credentialed agent from an anonymous bot. DataDome lists TAP and KYA trust-authority integrations on its pricing comparison.

Visa and Mastercard sign their agents' requests, so the question for a merchant is whether the CDN or bot manager already checks those signatures.

In practice

  1. Ask your CDN or bot manager whether it verifies Web Bot Auth. Cloudflare says TAP and Agent Pay both use it, with the tags agent-browser-auth or agent-payer-auth, and announced managed rulesets for registered agents; the current rule name is not confirmed in its documents.
  2. Compare vendors on the question. Akamai and Visa announced a TAP integration with Akamai's edge bot protection on 2025-12-17. DataDome lists TAP and KYA integrations on its pricing comparison, which is a vendor claim.
  3. Do not write a user-agent allow rule for these agents. A user-agent string can be copied by anyone, and the point of TAP is a cryptographic signature with timestamps that a merchant verifies.
  4. If you need to test, start from Visa's reference implementation on GitHub, which has agent-registry, cdn-proxy, merchant-backend and tap-agent folders.
  5. Expect no public list. The Verification Ledger found no public operator-level registry of registered agents for either programme.

What the sources say

Sources

Last reviewed 2026-11-04