X-Robots-Tag
Also called: X-Robots-Tag header, robots HTTP header
The X-Robots-Tag is an HTTP response header that tells search crawlers how to index or serve a URL, using the same directives as the robots meta tag (noindex, nofollow, and more). Because it lives in the server response, it can also control non-HTML files like PDFs, images, and videos.
Any directive you can put in a <meta name="robots"> tag also works as an X-Robots-Tag header: noindex, nofollow, none, nosnippet, noimageindex, max-snippet, max-image-preview, and unavailable_after. The difference is where it lives. A meta tag only exists inside HTML, so it cannot govern a PDF, an image, or a video file. The header rides on the HTTP response for any file type, which makes it the standard way to keep non-HTML assets out of the index.
You can also scope a rule to a single crawler by naming the user agent first:
HTTP/1.1 200 OK
X-Robots-Tag: googlebot: noindex, nofollow
X-Robots-Tag: otherbot: noindexRules written without a user agent apply to every crawler.
The robots.txt trap
There is one failure mode people hit constantly. If a URL is disallowed in robots.txt, Google never fetches the response, so it never sees the header. The noindex is ignored and the URL can still appear in results. For a directive to take effect, the page has to stay crawlable. Block a URL in robots.txt or noindex it in the header, not both at once. On a conflict between headers, the more restrictive rule wins (for example, nosnippet beats max-snippet:50).
How it affects your traffic
A single misplaced X-Robots-Tag can wipe pages out of Google. A stray noindex header left over from staging, or applied site-wide by a plugin or CDN rule, quietly drops URLs from the index and takes the clicks with them. The reverse hurts too: thin filter pages, internal PDFs, and duplicate assets that should carry a header often do not, so they burn crawl budget and dilute rankings. Auditing these headers across every file type is core Technical SEO work, and it is one of the faster ways to recover traffic that vanished with no obvious cause.
Get Technical SEO that moves the needle
We turn terms like this into ranked pages and qualified pipeline. Start with a free Initial SEO Strategy.