Crawler operator
Firecrawl
In short
Firecrawl turns pages into Markdown or JSON for AI applications, sending every request through proxies by default: basic ones in 26 countries, with enhanced proxies tried when those fail. The one address it publishes belongs to its API server, not its scrapers, and plans run from a free 1,000 credits a month to USD 83 for Standard billed annually.
- Type
- Crawler operator
- Homepage
- firecrawl.dev
- In one line
- Web scraping and crawling API for AI applications that matches robots.txt rules for the FirecrawlAgent token, routes every request through its own proxies, and warns that the token is not necessarily the User-Agent header it sends.
- Last read
- 2026-11-05
What Firecrawl does for this layer
From the vendor's own pages
Firecrawl reads robots.txt before it crawls a site and applies both the rules written for the FirecrawlAgent token and the rules for the * group. (docs.firecrawl.dev) Firecrawl's docs call FirecrawlAgent the token it matches in robots.txt, say not to assume it is also the User-Agent header on the fetch, and advise allow-listing Firecrawl with a robots.txt directive rather than a WAF rule on User-Agent. (docs.firecrawl.dev) Firecrawl publishes one IP address, 35.245.250.27, which its docs describe as the API server address that customers' own firewalls must allow to reach api.firecrawl.dev. (docs.firecrawl.dev) Firecrawl's proxy docs say all requests go through proxies by default, with US proxies when no location is set, basic proxies in 26 countries and enhanced proxies in the Netherlands and the United States. (docs.firecrawl.dev)
Firecrawl's default proxy mode is auto, which tries basic proxies first and retries with enhanced proxies on failure; the docs mark the proxy parameter as deprecated. (docs.firecrawl.dev) Firecrawl's pricing page lists a Free plan of 1,000 credits a month and Hobby at USD 19 a month (USD 16 billed annually) for 5,000 credits, where one credit is one page on a basic scrape or crawl. (firecrawl.dev)
Claims the vendor makes about itself
Firecrawl's pricing page carries a banner announcing "Alexandria" and a USD 75M Series B. (firecrawl.dev)
What our crawl saw
GETs for the Web Bot Auth key directory at www.firecrawl.dev and api.firecrawl.dev both returned HTTP 404 on 2026-11-05. (firecrawl.dev)
Products and plans
| Product | What it does | Price model | From | As of |
|---|---|---|---|---|
| Firecrawl API (scrape, crawl, map, search) | Turns pages into Markdown or JSON for AI applications. | credit subscription | USD 16 | 2026-11-05 |
Standards
| Standard | Status | Evidence |
|---|---|---|
| RFC 9309 (robots.txt) | Supports | Reads robots.txt before crawling and applies FirecrawlAgent and * rules. (docs.firecrawl.dev) |
| Web Bot Auth | No statement | No signing feature documented; key directory 404 (2026-11-05). |
| Published IP list | No statement | Only the API server IP is published, not crawler egress addresses. (docs.firecrawl.dev) |
Our checks
| Check | Result | Method | Date |
|---|---|---|---|
| Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | HTTP 404 | GET https://www.firecrawl.dev/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | HTTP 404, application/json | GET https://api.firecrawl.dev/.well-known/http-message-signatures-directory with the crawl user agent after a robots.txt check | 2026-11-05 |
What Firecrawl does not do
- Firecrawl does not promise a fixed User-Agent header that a WAF rule can match, by its own documentation.
- Firecrawl publishes no egress IP list or signature keys for its scraping traffic (checked 2026-11-05).
Related pages on this site
Sources
Last reviewed 2026-11-05