Glossary
RFC 9309
In short
The IETF Robots Exclusion Protocol standard, published September 2022, that defines robots.txt groups, Allow and Disallow rules, longest-match precedence and case-insensitive product-token matching. The block rates apply its matching rules, a group naming the token first and the * group otherwise, and the builder's output is tested against its grammar. Related: Block-rate method and robots.txt builder.
Definition
The IETF Robots Exclusion Protocol standard, published September 2022, that defines robots.txt groups, Allow and Disallow rules, longest-match precedence and case-insensitive product-token matching.
Sourced example
RFC 9309 says crawlers should match the product token in the User-agent line case-insensitively. Source, read 2026-11-04.
How it shows up here
The block rates apply its matching rules, a group naming the token first and the * group otherwise, and the builder's output is tested against its grammar.
Previous: Reverse DNS verification. Next: robots.txt. All 41 terms.
Sources
Last reviewed 2026-11-04