disallow
Also called: robots disallow, robots.txt disallow, disallow directive
Disallow is a robots.txt directive that tells search engine crawlers not to fetch URLs matching a given path, for example Disallow: /admin/. It controls crawling only, not indexing, so a disallowed URL can still appear in Google's results if other pages link to it.
The directive lives in your robots.txt file at the site root. Each rule pairs a User-agent line with one or more Disallow (and optional Allow) lines. The path value is relative to the domain root, must start with /, and is case-sensitive, so /Admin/ and /admin/ are treated as different paths.
Google supports two wildcards: * matches zero or more of any character, and $ marks the end of the URL. So Disallow: /*.pdf$ blocks every URL ending in .pdf. When an Allow and a Disallow rule both match a URL, Google uses the most specific rule based on path length, and when rules conflict it applies the least restrictive one.
Blocks crawling, not indexing
This is the part that trips people up. Google’s own docs say robots.txt “is not a mechanism for keeping a web page out of Google.” Disallow stops the crawler from fetching a page, but if other sites link to it, Google can still index the bare URL and show it without a snippet. To actually keep a page out of results, use a noindex tag or password protection.
There is a related trap: noindex and Disallow on the same URL cancel each other out. Google states that for noindex to work, the page “must not be blocked by a robots.txt file.” If it is blocked, the crawler never reaches the noindex rule, so the page can linger in the index. Let the page stay crawlable, apply noindex, then disallow it later once it has dropped out.
How it affects your traffic
A single stray Disallow: / can pull an entire site out of crawling, and misplaced rules quietly block CSS, JS, or whole product folders that Google needs to render and rank pages. Used well, a clean robots.txt protects crawl budget on large sites by steering bots away from faceted filters and infinite calendars. Our Technical SEO service audits your robots.txt line by line, checks it against what Google's crawler actually fetches, and fixes disallow rules that either leak private paths or strangle pages you want indexed.
Get Technical SEO that moves the needle
We turn terms like this into ranked pages and qualified pipeline. Start with a free Initial SEO Strategy.