---
title: "RFC 9309: definition | Agent Ready Crawlers"
description: "RFC 9309: the IETF Robots Exclusion Protocol standard of September 2022, which defines groups, Allow and Disallow rules and longest-match precedence."
canonical: https://agentreadycrawlers.com/glossary/rfc-9309
site: "Agent Ready Crawlers"
last_updated: 2026-11-04
section: "Glossary"
---

# RFC 9309: definition | Agent Ready Crawlers

> RFC 9309: the IETF Robots Exclusion Protocol standard of September 2022, which defines groups, Allow and Disallow rules and longest-match precedence.

Source: https://agentreadycrawlers.com/glossary/rfc-9309
Site: Agent Ready Crawlers
Last reviewed: 2026-11-04

Glossary

# RFC 9309

In short

The IETF Robots Exclusion Protocol standard, published September 2022, that defines robots.txt groups, Allow and Disallow rules, longest-match precedence and case-insensitive product-token matching. The block rates apply its matching rules, a group naming the token first and the \* group otherwise, and the builder's output is tested against its grammar. Related: Block-rate method and robots.txt builder.

## Definition

The IETF Robots Exclusion Protocol standard, published September 2022, that defines robots.txt groups, Allow and Disallow rules, longest-match precedence and case-insensitive product-token matching.

## Sourced example

RFC 9309 says crawlers should match the product token in the User-agent line case-insensitively. [Source](https://www.rfc-editor.org/rfc/rfc9309.html), read 2026-11-04.

## How it shows up here

The block rates apply its matching rules, a group naming the token first and the \* group otherwise, and the builder's output is tested against its grammar.

-   [Block-rate method](https://agentreadycrawlers.com/method/#block-rate-method)
-   [robots.txt builder](https://agentreadycrawlers.com/robots-builder/)

Previous: [Reverse DNS verification](https://agentreadycrawlers.com/glossary/reverse-dns-verification/). Next: [robots.txt](https://agentreadycrawlers.com/glossary/robots-txt/). [All 41 terms](https://agentreadycrawlers.com/glossary/).

## Sources

-   [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) read 2026-11-04

Last reviewed 2026-11-04