Agent Ready Crawlers

The field guide to every AI crawler and agent at the door, and the vendors who decide who gets in.

Search index crawler from Microsoft

Bingbot

In short

Bingbot is Microsoft's search index crawler. The token to write in robots.txt is bingbot. Verify it by reverse DNS, then a forward lookup. Hourly crawl control in Bing Webmaster Tools. Bingbot: robots.txt can be configured to tell Bing crawlers how to interact with a site; crawl rate is set by hour in Bing Webmaster Tools.

Operator
Microsoft Vendor profile: Microsoft
Purpose
Search index All search index rows. Builds a search index that sends visitors back. Refusing it removes pages from that product.
User-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36
robots.txt token
bingbot Matched case-insensitively (RFC 9309). The token and the user-agent string are different things.
Respects robots.txt

Yes, per operator

Microsoft: robots.txt can be configured to tell Bing crawlers how to interact with a site; crawl rate is set by hour in Bing Webmaster Tools. Source, read 2026-11-04.

Verification

Reverse DNS

Reverse-resolve the source address, then forward-resolve the name and require the same address. Documentation, read 2026-11-04.

IP ranges
bing.com/toolbox/bingbot.json
Rate behaviour
Hourly crawl control in Bing Webmaster Tools.
Block rate

3.1% of 5,938 top-10,000 sites with a parsed robots.txt disallow Bingbot at the root (182 sites). 0.4% name it; 2.7% refuse it only through a * group. Rank 30 of 31 tokens measured.

Across all 8,624 parsed robots.txt files in the crawl: 2.2% (0.2% named). Crawl of 2026-11-04; report.

Documentation
bing.com, read 2026-11-04.

Note. Verify Bingbot tool at bing.com/webmasters/verifybingbot. Training opt-out today is the NOARCHIVE meta tag; a robots.txt no-training preference is targeted for early 2027, per Cloudflare's 2026-09-15 post. bingbot.json creationTime 2024-01-03.

Allow and refuse lines

To allow bingbot

User-agent: bingbot
Allow: /

To refuse bingbot

User-agent: bingbot
Disallow: /

Refusing or allowing in robots.txt states a preference. Enforcement of a refusal belongs in your WAF; see how to verify the identity above.

Microsoft in the Verification Ledger

bingbot.json (creationTime 2024-01-03) and a verification tool; nothing for Browse with Copilot. All operators.

IP lists
Reverse DNS
Verify Bingbot tool and documented methods
Key directory
not published
Cloudflare
not checked
Visa TAP registry
no public operator-level registration found (checked 2026-11-04)
Mastercard Agent Pay
no public operator-level registration found (checked 2026-11-04)

In the news

Other Microsoft rows

Sources

Last reviewed 2026-11-04