Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Search index crawler from Huawei

PetalBot

In short

PetalBot is Huawei's search index crawler. The token to write in robots.txt is PetalBot. Verify it by reverse DNS, then a forward lookup. 14.1 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). On robots.txt, Huawei: PetalBot complies with the robots protocol; blocking it removes the site from Petal search.

Operator
Huawei Vendor profile: Huawei
Purpose
Search index All search index rows. Builds a search index that sends visitors back. Refusing it removes pages from that product.
User-agent string
Mozilla/5.0 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot)
robots.txt token
PetalBot Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.
Respects robots.txt

Yes, per operator

Huawei: PetalBot complies with the robots protocol; blocking it removes the site from Petal search. Source, read 2026-11-04.

Verification

Reverse DNS

Reverse-resolve the source address, then forward-resolve the name and require the same address. Documentation, read 2026-11-04.

IP ranges
Not published.
Rate behaviour
Huawei says it adjusts crawl volume by server capacity and site quality.
Block rate

14.1% of 5,938 top-10,000 sites with a parsed robots.txt disallow PetalBot at the root (835 sites). 10.3% name it; 3.8% refuse it only through a * group. Rank 14 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 10.9% (8.1% named). Crawl of 2026-11-04; report.

Documentation
webmaster.petalsearch.com, read 2026-11-04.

Note. Reverse DNS in petalsearch.com, then forward-confirm. Also feeds Huawei Assistant and AI Search.

Allow and refuse lines

To allow PetalBot

User-agent: PetalBot
Allow: /

To refuse PetalBot

User-agent: PetalBot
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

Huawei in the Verification Ledger

Reverse DNS only. All operators.

IP lists
publishes nothing
Reverse DNS
petalbot-*.petalsearch.com
Key directory
publishes nothing
Cloudflare
not checked
Visa TAP registry
no public operator-level registration found (checked 2026-11-04)
Mastercard Agent Pay
no public operator-level registration found (checked 2026-11-04)

Sources

Last reviewed 2026-11-04