---
title: "robots.txt: definition | Agent Ready Crawlers"
description: "robots.txt: A plain-text file at the root of a host that tells crawlers which paths they may fetch, grouped by user-agent token."
canonical: https://agentreadycrawlers.com/glossary/robots-txt
site: "Agent Ready Crawlers"
last_updated: 2026-11-04
section: "Glossary"
---

# robots.txt: definition | Agent Ready Crawlers

> robots.txt: A plain-text file at the root of a host that tells crawlers which paths they may fetch, grouped by user-agent token.

Source: https://agentreadycrawlers.com/glossary/robots-txt
Site: Agent Ready Crawlers
Last reviewed: 2026-11-04

Glossary

# robots.txt

In short

A plain-text file at the root of a host that tells crawlers which paths they may fetch, grouped by user-agent token. Standardised as RFC 9309; it states a preference and enforces nothing. The crawl parsed a robots.txt from 79.2 percent of the 10,886 domains that answered, and 67.4 percent of top-10,000 files carry a Sitemap line.

## Definition

A plain-text file at the root of a host that tells crawlers which paths they may fetch, grouped by user-agent token. Standardised as RFC 9309; it states a preference and enforces nothing.

## Sourced example

Anthropic documents "User-agent: ClaudeBot" followed by "Disallow: /" to block ClaudeBot site-wide, per subdomain. [Source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), read 2026-11-04.

## How it shows up here

The crawl parsed a robots.txt from 79.2 percent of the 10,886 domains that answered, and 67.4 percent of top-10,000 files carry a Sitemap line.

-   [How block rates are computed](https://agentreadycrawlers.com/method/#block-rate-method)
-   [robots.txt builder](https://agentreadycrawlers.com/robots-builder/)

Previous: [RFC 9309](https://agentreadycrawlers.com/glossary/rfc-9309/). Next: [Robots.txt token](https://agentreadycrawlers.com/glossary/robots-txt-token/). [All 41 terms](https://agentreadycrawlers.com/glossary/).

## Sources

-   [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) read 2026-11-04

Last reviewed 2026-11-04