robots.txt
Also called: robots exclusion protocol
robots.txt is a plain-text file placed at the root of a site that tells search-engine crawlers which URLs they may or may not request. It is meant to manage crawler traffic, not to keep pages out of the index.
What robots.txt actually controls
robots.txt governs crawling, not indexing. When a crawler like Googlebot
requests a site, it reads /robots.txt first and obeys the Allow and Disallow
rules for its user-agent before fetching anything else. The file lives at the
domain root (https://example.com/robots.txt) and applies only to that host and
protocol.
Since 2022 the format is a published standard, RFC 9309,
which is why every major search engine parses it the same way. A typical file
groups rules by User-agent, lists paths to Disallow, and points to the XML
sitemap:
User-agent: *
Disallow: /cart/
Disallow: /search
Sitemap: https://example.com/sitemap.xmlThe mistake that costs traffic
The most common misconception is that Disallow removes a page from Google. It
does not. Blocking a URL stops Google from crawling its content, but if other
pages link to it, Google can still index the URL itself and show it in results
without a description. To actually keep a page out of the index, let it be crawled
and add a noindex rule. A page blocked by robots.txt can never be read, so
Google never sees the noindex.
Because the file is read top-to-bottom and applies site-wide, a single stray
Disallow: / pushed to production can deindex an entire site over the following
crawls. Any change to robots.txt belongs in the same review lane as a deploy.
How it affects your traffic
A wrong disallow rule is one of the fastest ways to lose traffic: block a section by mistake and Google stops recrawling it, so updates never register and, over time, pages drop. The opposite error is just as common, teams block a page in robots.txt to hide it, then it still shows up in results as a bare URL because blocking crawling is not the same as removing it from the index.
Get Technical SEO that moves the needle
We turn terms like this into ranked pages and qualified pipeline. Start with a free Initial SEO Strategy.