Crawler operator
Microsoft
In short
From 1 February 2027 Bingbot will honour a shared training preference written into robots.txt, while a page-level grounding preference, set with a meta tag or header, can keep content out of services sold to third parties. Neither signal governs crawl access or sets up payment, and bingbot.json lists 29 IPv4 prefixes.
- Type
- Crawler operator
- Homepage
- microsoft.com
- In one line
- Microsoft crawls for Bing and Copilot with Bingbot, publishes bingbot.json, will read the shared AI training preference and its own page-level grounding preference from 2027-02-01, and runs a Publisher Content Marketplace for licensed grounding.
- Last read
- 2026-11-05
What Microsoft does for this layer
From the vendor's own pages
Microsoft announced on 2026-10-30 that from 2027-02-01 it will honour the shared IETF AI Training preference in robots.txt, written as a group for User-Agent: __usage_ai-training. (blogs.bing.com, dated 2026-10-30) Microsoft adds a page-level AI Grounding preference with three values, Allow, DisallowExtended and Disallow, where DisallowExtended keeps grounding to Microsoft-run experiences such as Copilot and Microsoft 365 and excludes grounding services sold to third parties. (blogs.bing.com, dated 2026-10-30) Microsoft says NoArchive and NoCache stop being read as training or grounding preferences after 2027-03-01, and asks sites that rely on them to publish the new preferences before then. (blogs.bing.com, dated 2027-03-01) Microsoft says both preferences default to Allow, do not change eligibility for traditional Bing results, and do not set up licensing or payment. (blogs.bing.com)
Microsoft says the Bing search syndication API returns links for traditional search and is not affected by the grounding preference. (blogs.bing.com) Microsoft's page-level syntax is a meta tag named Bingbot or an X-Robots-Tag header, for example content="ai-training:disallow" or ai-grounding:disallowextended. (blogs.bing.com) Microsoft says its training and grounding settings govern use of content, not crawl access, so an Allow does not override a robots.txt rule that blocks Bingbot. (blogs.bing.com) Microsoft opens a temporary grounding preference in Bing Webmaster Tools on 2026-12-15 for sites still using NoArchive or NoCache, and any such temporary preference expires on 2027-09-30. (blogs.bing.com, dated 2026-12-15)
Microsoft launched the Publisher Content Marketplace on 2026-02-03, a pilot in which publishers set licensing and usage terms, AI builders license content for grounding, and publishers are "paid on delivered value". (about.ads.microsoft.com, dated 2026-02-03)
Claims the vendor makes about itself
Microsoft says it co-designed the marketplace with publishers including The Associated Press, Condé Nast, Hearst Magazines, People Inc, USA TODAY Co. and Vox Media, began with grounding in Microsoft Copilot, and is onboarding Yahoo as a demand partner. (about.ads.microsoft.com, dated 2026-02-03)
What our crawl saw
Microsoft's bingbot.json listed 29 IPv4 prefixes with a creationTime of 2024-01-03 when fetched on 2026-11-05. (bing.com, dated 2024-01-03) Neither www.bing.com nor www.microsoft.com served a Web Bot Auth key directory on 2026-11-05; both returned HTTP 404. (bing.com)
Products and plans
| Product | What it does | Price model | From | As of |
|---|---|---|---|---|
| Bingbot | Crawler for Bing search and the web grounding behind Copilot and Microsoft 365. | |||
| AI Training and AI Grounding preferences | Robots.txt and page-level controls read from 2027-02-01 that separate training use from grounding use. | |||
| Publisher Content Marketplace | Pilot marketplace where publishers set terms and AI builders license content for grounding, with usage-based reporting. | |||
Standards
| Standard | Status | Evidence |
|---|---|---|
| RFC 9309 (robots.txt) | Supports | Microsoft says an Allow preference does not override a robots.txt rule blocking Bingbot, and reads the AI Training preference from robots.txt groups from 2027-02-01. (blogs.bing.com) |
| Web Bot Auth | No statement | No key directory on www.bing.com or www.microsoft.com (observed 404, 2026-11-05). |
| Content Signals | No statement | Microsoft's October 2026 post cites the IETF AI Preferences draft, not Content-Signals. |
| Published IP list | Supports | bingbot.json published at bing.com/toolbox (29 IPv4 prefixes observed 2026-11-05). (bing.com) |
| Reverse DNS check | No statement | Bing's crawler help pages are JavaScript-only and could not be read with the crawl user agent (2026-11-05). |
Our checks
| Check | Result | Method | Date |
|---|---|---|---|
| Homepage fetch | HTTP 200 | GET with the crawl user agent after a robots.txt check | 2026-11-05 |
| Web Bot Auth key directory | 404 on www.bing.com and www.microsoft.com | GET /.well-known/http-message-signatures-directory with the crawl user agent | 2026-11-05 |
| bingbot.json | 29 IPv4 prefixes, creationTime 2024-01-03 | GET and count prefixes | 2026-11-05 |
What Microsoft does not do
- Microsoft accepts the AI Grounding preference only at page level; it cannot be set in robots.txt.
- Microsoft's October 2026 post names no separate Copilot user agent, so Copilot grounding is controlled through the preferences rather than by blocking a token.
- Microsoft's preferences take effect on 2027-02-01, so a robots.txt training rule written today is not yet read by Bingbot as a training preference.
Reading notes
Bing's webmaster crawler help pages render only with JavaScript, so Bingbot's reverse DNS and token details were not re-read from them; see the Bingbot row.
Related pages on this site
Sources
Last reviewed 2026-11-05