# RFC 9309: Robots Exclusion Protocol - RFC Editor drillable.com › rfc-editor-org › rfc9309. As of the capture of 13 Sep 2026; the index built 13 Sep 2026, 22:06 UTC · [cites, never asserts](/standing). Source: https://www.rfc-editor.org/info/rfc9309/ · quotes only, no licence recorded. Search `/rfc-editor-org/rfc9309/?q=…` · MCP https://mcp.drillable.com · [html](/rfc-editor-org/rfc9309.html) · [markdown](/rfc-editor-org/rfc9309.md) · [json](/rfc-editor-org/rfc9309.json) · [discovery](/.well-known/drillable.json) · [guide](/llms.txt) ## Records read from it (33) By drillable.com ([llm-extract/3](/standing#methods)); scored precision 1.00, recall 0.24 on the harness's last sample of 37. ### [robots.txt](/?subject=https%3A%2F%2Fdrillable.com%2Fe%2Frfc-editor-org%2Frfc9309%2Frobots.txt) (16 records) - **rule** = If no match is found amongst the rules in a group for a matching user-agent or there are no rules in the group, the URI is allowed. · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=9074&end=9205) · [sha256:e2b36f4a2fa0ce0f…](/r/sha256:e2b36f4a2fa0ce0f0958ca3e1c8efdad71a0783c9e809bb92ee1dcae2fe1675a) - **rule** = The /robots.txt URI is implicitly allowed. · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=9206&end=9248) · [sha256:72c1da48519d5a22…](/r/sha256:72c1da48519d5a22ab2100d7ea2bb52b61db5570aa94e6d27c3c35a720fdcf7a) - **rule** = Parsing of other records MUST NOT interfere with the parsing of explicitly defined records in Section 2. · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=13434&end=13538) · [sha256:4af8e71ff4e276fa…](/r/sha256:4af8e71ff4e276fa3fd340d5d8044946b18aa6c7a37fe6c65e716ed5e7d49586) - **io** = scheme:[//authority]/robots.txt · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=13916&end=13949) · [sha256:b6b7620423c06f90…](/r/sha256:b6b7620423c06f908db6ba179515bf1ff7d5414ebb1338f792752469fe76e71a) - **rule** = If a robots.txt file is reached within five consecutive redirects, the robots.txt file MUST be fetched, parsed, and its rules followed in the context of the initial authority. · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=14531&end=14706) · [sha256:78997700cdaab3f9…](/r/sha256:78997700cdaab3f93df1ca5932af17019cfb64f5f8ccea75b434799f89279471) - **effect** = Listing paths in the robots.txt file exposes them publicly and thus makes the paths discoverable. · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=16426&end=16523) · [sha256:bb500e692d85d688…](/r/sha256:bb500e692d85d6888a12d4124a4371ff23b02fcbfabb243538b6ecf87a895ea3) - **rule · definition** = A line with a key-value pair that defines how a crawler may access URIs. · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=3033&end=3105) · [sha256:99f6972846e63a8f…](/r/sha256:99f6972846e63a8fbd40c379789d4af0ccf3dfcd7159520d2d7a52112a99dbd2) - **group · definition** = One or more user-agent lines that are followed by one or more rules. · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=3136&end=3204) · [sha256:38a3b2ddb553d2d0…](/r/sha256:38a3b2ddb553d2d0714fcfc8af6c2efacac2e06ba0fdeac1b417a4385736329f) - **group · rule** = If there is more than one group matching the user-agent, the matching groups' rules MUST be combined into one group and parsed according to Section 2.2.2. · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=6430&end=6584) · [sha256:ea1ab9cf45522e46…](/r/sha256:ea1ab9cf45522e461f9357e65cfbb6f8df49b14cd3c99319a9be8783b675b3ce) - **group · rule** = Duplicate rules in a group MAY be deduplicated. · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=8931&end=8978) · [sha256:5e7b3cebcf068e92…](/r/sha256:5e7b3cebcf068e9209681f41d0864e6bb0c6d4818f968f908846978bf8306378) - **matching · rule** = The most specific match found MUST be used. · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=8824&end=8867) · [sha256:dc001855517de085…](/r/sha256:dc001855517de08514ed7ed555d120947f74759227c6db9900036364dc801cbb) - **matching · rule** = The match evaluates positively if and only if the end of the path from the rule is reached before a difference in octets is encountered. · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=9700&end=9836) · [sha256:8185c34587ff96e9…](/r/sha256:8185c34587ff96e9bb55226b871f8ad639d943126e2899f0ab59d31d190a7216) - **unavailable status · range** = 400-499 · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=15059&end=15101) · [sha256:97438a683dbb3e10…](/r/sha256:97438a683dbb3e108fe5c0eeb29b433a6929f31c11332dbdc880ea2335412d88) - **unreachable status · range** = 500-599 · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=15486&end=15551) · [sha256:b2fe8d6fc5593cd5…](/r/sha256:b2fe8d6fc5593cd5f9cdcedab3b4c65286dd736f6df837689e815961b6bf4590) - **unreachable status · period** = 30 days · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=15630&end=15650) · [sha256:ad598045b1672e90…](/r/sha256:ad598045b1672e90f046ec1e040eab29f5be5752a375e2303d2728ea803e92dc) - **limits · size** = 500 kibibytes · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=16252&end=16307) · [sha256:f72d171700b42723…](/r/sha256:f72d171700b427234eb7575c3092adab91a691ff65910a7c0d23622dabb92f25) ### [rfc9309](/?subject=https%3A%2F%2Fdrillable.com%2Fe%2Frfc-editor-org%2Frfc9309%2Frfc9309) (7 records) - **author** = M. Koster · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=135&end=145) · [sha256:d228f7f4ce32cdae…](/r/sha256:d228f7f4ce32cdaecc8bf97597d23dd5bf81eab28d62a1c83545b9086ff0c64d) - **author** = G. Illyes · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=147&end=157) · [sha256:2eae0183d0473bdf…](/r/sha256:2eae0183d0473bdf85adbd67d5ff4bdd414ea078f822943f56cb36f6bd198aae) - **author** = H. Zeller · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=159&end=169) · [sha256:ed6517669e5c8ec2…](/r/sha256:ed6517669e5c8ec25d5f03245d53d2bc34de003fe09e2f202755032d54b53285) - **author** = L. Sassman · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=171&end=181) · [sha256:1029fd4095b6366b…](/r/sha256:1029fd4095b6366b1d34228b8b3f542ab6350bad5b5e6af4c87989cb32314028) - **status** = Proposed Standard · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=183&end=200) · [sha256:8cae455ff5cd3701…](/r/sha256:8cae455ff5cd370187e9d8b16d2de74e60b42a60be162d22d6b7aac32c5b1b1a) - **type** = Internet Standards Track document · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=615&end=660) · [sha256:d97b373bb976aab0…](/r/sha256:d97b373bb976aab02f6b76bf49b1fab9bd787701d3b061283337fd832f63c57a) - **date** = 2022 · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=1174&end=1255) · [sha256:698b31ab80147b96…](/r/sha256:698b31ab80147b96a6abb6d3d32843e307c1cbf2b339710af144046c0d1ba031) ### [crawler](/?subject=https%3A%2F%2Fdrillable.com%2Fe%2Frfc-editor-org%2Frfc9309%2Fcrawler) (4 records) - **rule** = If the crawler successfully downloads the robots.txt file, the crawler MUST follow the parseable rules. · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=14137&end=14240) · [sha256:5d6676a6d614ab94…](/r/sha256:5d6676a6d614ab941a535abc452f043d346f77bffa986f4460d1c74d9015aa3f) - **rule** = If there are more than five consecutive redirects, crawlers MAY assume that the robots.txt file is unavailable. · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=14710&end=14821) · [sha256:f499cc37776c8888…](/r/sha256:f499cc37776c8888d1a7654cc901538bd2fd5dc2f629d1e5a4748e9d71fad07d) - **rule** = If a server status code indicates that the robots.txt file is unavailable to the crawler, then the crawler MAY access any resources on the server. · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=15106&end=15252) · [sha256:40e107d17be3f8e4…](/r/sha256:40e107d17be3f8e427686dbad1ce3ab9b3afcc79c40b5e9331eb0f45b3b8298c) - **rule** = If the robots.txt file is unreachable due to server or network errors, this means the robots.txt file is undefined and the crawler MUST assume complete disallow. · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=15287&end=15448) · [sha256:bb489b9e00aaaee4…](/r/sha256:bb489b9e00aaaee499449a2f933980285ad625458516767c65f662b00ad5e64a) ### [robots exclusion protocol](/?subject=https%3A%2F%2Fdrillable.com%2Fe%2Frfc-editor-org%2Frfc9309%2Frobots-exclusion-protocol) (2 records) - **date** = 1994 · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=287&end=331) · [sha256:12045e58fb6cd47f…](/r/sha256:12045e58fb6cd47fd093b5ca479663a059fb2fa532e3fb2f181e0ca546a4e199) - **author** = Martijn Koster · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=287&end=331) · [sha256:f7206e43431f3abc…](/r/sha256:f7206e43431f3abcb80e51f037f3a0be099a7349617e979aa133e9a28317aced) ### [gary illyes](/?subject=https%3A%2F%2Fdrillable.com%2Fe%2Frfc-editor-org%2Frfc9309%2Fgary-illyes) (1 record) - **email** = garyillyes@google.com · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=21662&end=21690) · [sha256:17da14ce325aa159…](/r/sha256:17da14ce325aa1596dff148462a4fb43671c3f014138bc72f2bfbe7ff23f8488) ### [henner zeller](/?subject=https%3A%2F%2Fdrillable.com%2Fe%2Frfc-editor-org%2Frfc9309%2Fhenner-zeller) (1 record) - **email** = henner@google.com · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=21794&end=21818) · [sha256:571819460fe14c89…](/r/sha256:571819460fe14c89e733d8e706c0332bc53536854d678f642747a0eeeccb6790) ### [lizzi sassman](/?subject=https%3A%2F%2Fdrillable.com%2Fe%2Frfc-editor-org%2Frfc9309%2Flizzi-sassman) (1 record) - **email** = lizzi@google.com · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=21902&end=21925) · [sha256:55602db6916f566e…](/r/sha256:55602db6916f566e1960c4ade5be42d567b3b034e9dc508670946f6b046bbfb1) ### [martijn koster](/?subject=https%3A%2F%2Fdrillable.com%2Fe%2Frfc-editor-org%2Frfc9309%2Fmartijn-koster) (1 record) - **email** = m.koster@greenhills.co.uk · [quote](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97?trace&start=21548&end=21580) · [sha256:06dff7c503db6d69…](/r/sha256:06dff7c503db6d6929d4fc4492d5f1fcf737de63d5675393b23baf3da898a038) ## Captures (1) - 13 Sep 2026, 20:26 UTC · https://www.rfc-editor.org/info/rfc9309/ · quotes only (no licence recorded) · 0 revisits · [text](/r/sha256:4d2cbe34173d0888ef34459e4083ef215d85e1dbe82ca24ab41f36eeacd6ea97) (drillable-html-text/1, 22849 bytes) · capture [sha256:36e4f173f39d6fee…](/r/sha256:36e4f173f39d6feea1e20008863ea83ab8eac60df23f8de44f4f47ad202b8da1) A capture is what the source served at that moment; a later change at the source is a new capture, never an edit to this one. --- [standing](/standing) · Drillable™ is a trademark of Drillable LLC.