Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Bot from Firecrawl

FirecrawlAgent

In short

FirecrawlAgent is Firecrawl's bot, filed under Other. The token to write in robots.txt is FirecrawlAgent. The operator publishes nothing a site can check. The crawl of 2026-11-04 did not measure FirecrawlAgent. Firecrawl advises allowlisting with robots.txt rather than a WAF rule on User-Agent. FirecrawlAgent reads robots.txt before it crawls and applies rules for FirecrawlAgent and for *.

Operator
Firecrawl Vendor profile: Firecrawl
Purpose
Other All other rows. Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes.
User-agent string
No exact string is published. What the operator or a third party says: Not fixed: Firecrawl says do not assume FirecrawlAgent is the User-Agent header on the fetch
robots.txt token
FirecrawlAgent Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.
Respects robots.txt

Yes, per operator

Firecrawl: reads robots.txt before it crawls and applies rules for FirecrawlAgent and for *. Source, read 2026-11-04.

Verification

Nothing published

The operator publishes no list, reverse DNS pattern or key directory a site owner can check. Documentation, read 2026-11-04.

IP ranges
Not published.
Rate behaviour
Set by the customer job.
Block rate

Not measured: the crawl checked 31 AI and search tokens, and FirecrawlAgent is not one of them.

Documentation
docs.firecrawl.dev, read 2026-11-04.

Note. Firecrawl advises allowlisting with robots.txt rather than a WAF rule on User-Agent. GitHub issue #1169 records complaints that the token never appears in logs.

Allow and refuse lines

To allow FirecrawlAgent

User-agent: FirecrawlAgent
Allow: /

To refuse FirecrawlAgent

User-agent: FirecrawlAgent
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

Firecrawl in the Verification Ledger

Robots.txt token only; Firecrawl advises against UA-based allowlisting. All operators.

IP lists
publishes nothing
Reverse DNS
publishes nothing
Key directory
publishes nothing
Cloudflare
not checked
Visa TAP registry
no public operator-level registration found (checked 2026-11-04)
Mastercard Agent Pay
no public operator-level registration found (checked 2026-11-04)

Sources

Last reviewed 2026-11-04