The noindex rule tells search engines not to show a page in their results. Put <meta name="robots" content="noindex"> in the page head, or send an X-Robots-Tag: noindex header for files like PDFs. The page must stay crawlable: if robots.txt blocks it, Google never sees the rule.
| Meta robots tag | X-Robots-Tag header | |
|---|---|---|
| Where it goes | Inside <head> of an HTML page | In the server's HTTP response |
| Works for | HTML pages | Any file: PDFs, images, HTML |
| Who can add it | Anyone who can edit the page | Needs server or host settings |
| Example | <meta name="robots" content="noindex"> | X-Robots-Tag: noindex |
Updated
Business sites usually need noindex for a few pages: thank-you pages after a form, internal search results, staging copies, and downloadable files meant only for customers. Tick the rules you want below and copy the meta tag, the header, or the Apache and Nginx lines that send it.
Which crawlers: All search engines (robots), Google only (googlebot)
Stop showing after this date (optional)
<meta name="robots" content="noindex">X-Robots-Tag: noindex<Files ~ "\.pdf$">
Header set X-Robots-Tag "noindex"
</Files>location ~* \.pdf$ {
add_header X-Robots-Tag "noindex";
}The server examples match PDF files; change the pattern to match the files you need.
Frameworks render the same meta tag. In the Next.js App Router, the metadata object has a robots field.
Exported from a page or layout file; Next.js writes the robots meta tag into the head.
export const metadata = {
robots: { index: false, follow: true },
};robots.txt controls crawling: whether a bot may fetch a page at all. Noindex controls indexing: whether a fetched page may appear in results. Combining them on one page backfires, because the block stops Google from reading the noindex.
The order that works is to allow crawling, add noindex, wait for the page to drop out, and only then block it in robots.txt if you still want to save crawl visits.
Swap robots for googlebot in the name attribute to address only Google. In the header form, put the crawler name and a colon before the rules, for example X-Robots-Tag: googlebot: nofollow.
Either Google has not recrawled it yet, or robots.txt blocks it. Google's documentation says the page must not be blocked by robots.txt for noindex to work, because a blocked page is never fetched and the rule is never read.
Noindex keeps the page itself out of results. Nofollow asks crawlers not to follow the links on the page. The rule none is shorthand for both together.
No. Google's robots meta tag documentation says the header, the user agent name and the values are not case sensitive.
Google supports it fully. Its documentation notes that some other search engines may interpret the rule differently, so a page could still appear elsewhere.
Google applies the more restrictive one. Its example: if a page has both max-snippet:50 and nosnippet, nosnippet wins.
Related: Robots.txt generator, Sitemap and robots.txt examples, Robots.txt tester, Remove a URL from Google, Meta tag generator.
Start for free or see all free tools. We.Inc is free to start, with no card needed. AI building is on paid plans from $20 a month.
We.Inc is an AI-powered website builder you can resell under your own brand. Launch a branded client dashboard, bill on Stripe Connect, and deliver AI-generated websites in minutes. White-label plans start at $99 a month for 25 client sites, with a 7-day free trial and no per-site fees.