Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Bot from ByteDance

Bytespider

In short

Bytespider is ByteDance's bot, filed under Other. The token to write in robots.txt is Bytespider. The operator publishes nothing a site can check. 19.2 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Verisoul cites Cloudflare Radar at about 3 percent of global crawler traffic in September 2026.

Operator
ByteDance
Purpose
Other All other rows. Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes.
User-agent string
No exact string is published. What the operator or a third party says: Contains "Bytespider"; ByteDance publishes no string
robots.txt token
Bytespider Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.
Respects robots.txt

No documentation

operator publishes no bot documentation (checked 2026-11-04). No ByteDance page documents its purpose, IP ranges or robots.txt handling. Source, read 2026-11-04.

Operator publishes no bot documentation (checked 2026-11-04).

Verification

Nothing published

The operator publishes no list, reverse DNS pattern or key directory a site owner can check. Documentation, read 2026-11-04.

IP ranges
Not published.
Rate behaviour
Not published.
Block rate

19.2% of 5,938 top-10,000 sites with a parsed robots.txt disallow Bytespider at the root (1,141 sites). 15.4% name it; 3.8% refuse it only through a * group. Rank 2 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 15.0% (12.3% named). Crawl of 2026-11-04; report.

Documentation
Operator publishes no bot documentation (checked 2026-11-04). Closest page read.

Note. Verisoul cites Cloudflare Radar at about 3 percent of global crawler traffic in September 2026. Public reports of it ignoring robots.txt exist but are not operator statements.

Allow and refuse lines

To allow Bytespider

User-agent: Bytespider
Allow: /

To refuse Bytespider

User-agent: Bytespider
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

ByteDance in the Verification Ledger

Publishes nothing. All operators.

IP lists
publishes nothing
Reverse DNS
publishes nothing
Key directory
publishes nothing
Cloudflare
not found on Radar (slug bytespider returned 404, 2026-11-04)
Visa TAP registry
no public operator-level registration found (checked 2026-11-04)
Mastercard Agent Pay
no public operator-level registration found (checked 2026-11-04)

Sources

Last reviewed 2026-11-04