robots meta tag
Also called: meta robots tag, noindex tag, meta robots
The robots meta tag is an HTML element in a page's <head> that gives search engines page-level indexing and serving instructions, such as noindex (keep this page out of results) or nofollow (don't follow its links). Unlike site-wide robots.txt, it controls one specific page.
How it works
The tag lives in the document <head> and applies to a single page:
<meta name="robots" content="noindex, nofollow">name="robots" speaks to every crawler; swap in googlebot to target only Google. The directives you reach for most: noindex (keep the page out of results), nofollow (don’t follow or pass signals through its links), none (equals noindex, nofollow), plus nosnippet, max-snippet:[number], and max-image-preview:[setting] for previews. Google now lists noarchive and nocache as no longer used.
For files you can’t add HTML to (PDFs, images), the X-Robots-Tag HTTP header carries the same directives at the response level and can be applied across a whole site.
One trap costs people indexing every week: noindex only works if Google can crawl the page. If you also block that URL in robots.txt, Googlebot never fetches it, never sees the noindex, and the URL can linger in results. To deindex, allow crawling and use noindex, not a robots.txt block.
The AI angle: per Google’s own robots specs, nosnippet stops a page’s content being used as direct input to AI Overviews and AI Mode, and max-snippet caps how much can be used. A noindex page drops out of generative answers entirely, since a page ineligible for Search can’t surface there either.
How it affects your traffic
Misused robots directives are one of the fastest ways to erase organic traffic. A single stray noindex shipped in a page template, or a noindex page that is also blocked in robots.txt, can quietly pull money pages out of Google and out of AI answers without any obvious error. Our Technical SEO work audits every indexing directive across your site, checks meta robots against X-Robots-Tag and robots.txt for conflicts, and makes sure the pages you want ranked stay crawlable and indexable while thin or duplicate URLs get cleanly deindexed.
Get Technical SEO that moves the needle
We turn terms like this into ranked pages and qualified pipeline. Start with a free Initial SEO Strategy.