---
title: "Bytespider: user agent, token, block rate | Agent Ready Crawlers"
description: "Bytespider from ByteDance: robots.txt token Bytespider, other purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04."
canonical: https://agentreadycrawlers.com/crawlers/bytespider
site: "Agent Ready Crawlers"
last_updated: 2026-11-04
section: "Crawlers"
---

# Bytespider: user agent, token, block rate | Agent Ready Crawlers

> Bytespider from ByteDance: robots.txt token Bytespider, other purpose and what a site can check; no exact user-agent string is published. Read 2026-11-04.

Source: https://agentreadycrawlers.com/crawlers/bytespider
Site: Agent Ready Crawlers
Last reviewed: 2026-11-04

Bot from ByteDance

# Bytespider

In short

Bytespider is ByteDance's bot, filed under Other. The token to write in robots.txt is Bytespider. The operator publishes nothing a site can check. 19.2 percent of 5,938 top-10,000 sites with a parsed robots.txt disallow it at the root (crawl of 2026-11-04). Verisoul cites Cloudflare Radar at about 3 percent of global crawler traffic in September 2026.

Operator

ByteDance

Purpose

Other [All other rows](https://agentreadycrawlers.com/crawlers/purpose/other/). Link previews, ad checks, dataset crawls, SEO tools and legacy tokens that fit none of the five purposes.

User-agent string

No exact string is published. What the operator or a third party says: Contains "Bytespider"; ByteDance publishes no string

robots.txt token

`Bytespider` Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.

Respects robots.txt

No documentation

operator publishes no bot documentation (checked 2026-11-04). No ByteDance page documents its purpose, IP ranges or robots.txt handling. [Source](https://www.centinelanalytica.com/bots/bytespider), read 2026-11-04.

Operator publishes no bot documentation (checked 2026-11-04).

Verification

Nothing published

The operator publishes no list, reverse DNS pattern or key directory a site owner can check. [Documentation](https://www.centinelanalytica.com/bots/bytespider), read 2026-11-04.

IP ranges

Not published.

Rate behaviour

Not published.

Block rate

**19.2%** of 5,938 top-10,000 sites with a parsed robots.txt disallow `Bytespider` at the root (1,141 sites). 15.4% name it; 3.8% refuse it only through a \* group. Rank 2 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 15.0% (12.3% named). Crawl of 2026-11-04; [report](https://agentreadycrawlers.com/block-rates/).

Documentation

Operator publishes no bot documentation (checked 2026-11-04). [Closest page read](https://www.centinelanalytica.com/bots/bytespider).

Note. Verisoul cites Cloudflare Radar at about 3 percent of global crawler traffic in September 2026. Public reports of it ignoring robots.txt exist but are not operator statements.

## Allow and refuse lines

### To allow Bytespider

User-agent: Bytespider
Allow: /

### To refuse Bytespider

User-agent: Bytespider
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

## ByteDance in the Verification Ledger

Publishes nothing. [All operators](https://agentreadycrawlers.com/verification/).

IP lists

publishes nothing

Reverse DNS

publishes nothing

Key directory

publishes nothing

Cloudflare

not found on Radar (slug bytespider returned 404, 2026-11-04)

Visa TAP registry

no public operator-level registration found (checked 2026-11-04)

Mastercard Agent Pay

no public operator-level registration found (checked 2026-11-04)

## Sources

-   [centinelanalytica.com: bytespider](https://www.centinelanalytica.com/bots/bytespider) read 2026-11-04
-   [Verisoul: Bytespider (cites Cloudflare Radar)](https://www.verisoul.ai/agents/bytespider) read 2026-11-04

Last reviewed 2026-11-04