robots exclusion protocol
Also called: REP, robots.txt standard
The Robots Exclusion Protocol (REP) is the standard behind robots.txt: rules a site owner publishes to tell automated crawlers which URLs they may or may not fetch. Formalized as IETF RFC 9309 in 2022 (from Martijn Koster's 1994 method), it governs crawling only, not indexing, and is advisory, not an access control.
The REP is the shared vocabulary crawlers and site owners use to negotiate access. A robots.txt file at your domain root holds user-agent blocks with Allow and Disallow rules, plus wildcards (* matches any characters, $ marks the end of a URL). RFC 9309 pins down the parsing details: a crawler should not reuse a cached copy for more than 24 hours, and must parse at least the first 500 KiB of the file.
User-agent: *
Disallow: /search
Disallow: /*?filter=Crawl control, not access control
The protocol’s biggest trap is treating it like a lock. It is not one. RFC 9309 states plainly that “these rules are not a form of access authorization,” and Google warns that a disallowed URL can still appear in search results if other pages link to it. Blocking a page in robots.txt only stops the fetch, so Google may index the bare URL with no snippet. To actually keep a page out of results, use a noindex meta tag (which requires the page to stay crawlable so the tag can be read) or password-protect it. Note that listing a “secret” path in robots.txt makes it publicly discoverable, because the file itself is world-readable.
Good REP hygiene means blocking crawl traps (faceted filters, infinite calendars, internal search results) while keeping your important pages and their noindex tags reachable.
How it affects your traffic
Misused robots.txt is one of the quietest ways to lose traffic. A single overbroad Disallow can wall off entire sections from Google, while blocking a page you meant to deindex just strands its URL in the index with no snippet, since the crawler can never read the noindex. On large sites, tight REP rules also steer crawl budget toward pages that actually earn rankings instead of parameter noise. Auditing and fixing these rules is core to our Technical SEO work, where crawl directives are checked before content or links.
Get Technical SEO that moves the needle
We turn terms like this into ranked pages and qualified pipeline. Start with a free Initial SEO Strategy.