Robots Meta Tag Generator

A robots meta tag tells search engines whether to show a page in results, whether to follow its links, and how large a snippet or image preview they may use. Choose the rules here and copy either the HTML tag for the page head or the matching X-Robots-Tag line for an HTTP header, which works for PDFs and images too.

What you enter

  • Show in search results
  • Follow links on the page
  • No cached copy (noarchive)
  • No translation offer (notranslate)
  • Do not index images on the page (noimageindex)
  • Text snippet
  • Snippet characters
  • Image preview
  • Video preview
  • Video seconds
  • Crawler
  • Format

How it works

  1. Choose whether the page may appear in search results (index or noindex) and whether its links may be followed.
  2. Set the preview rules: the longest text snippet, the largest image preview and the video preview length. Leave them on default if you have no reason to limit them.
  3. Pick a format: the meta tag for HTML pages, or the HTTP header for files and server settings.
  4. Paste the tag inside the head of the page, or add the header in your server or hosting configuration.

What each directive means

index and follow are the defaults, so a tag of index, follow changes nothing. noindex keeps the page out of results, nofollow stops crawlers following its links, and none is a short form of both. noarchive asks for no cached copy, and notranslate asks search engines not to offer a translated version of the result.

The preview directives control how a result looks. max-snippet sets the text length, max-image-preview takes none, standard or large, and max-video-preview sets the number of seconds of video preview, with -1 meaning no limit. These only shape the result, they do not affect whether the page is shown.

Mistakes that hide whole sites

The most common problem is a noindex added to a development or staging site and copied to the live site at launch. Website builders and content systems often have a setting such as discourage search engines, which adds this tag for you. Check it after every launch.

The second is mixing up crawling and indexing. A page blocked in robots.txt can still appear in results as a bare address if other sites link to it, because the crawler never reads the tag that says otherwise. Let the page be crawled, use noindex, and only block crawling once it has dropped out.

Tips

  • Do not block a noindex page in robots.txt. If crawlers cannot fetch the page, they never see the noindex tag, and the address can stay in results.
  • A page with no robots tag is treated as index, follow. You only need the tag when you want something different.
  • Use noindex for thank-you pages, internal search results and staging copies, not for pages you want people to find.
  • Check the live page after publishing. A noindex left on from a staging site is a common reason a new site never shows up in search.
  • max-image-preview:large allows big images in search features and is a sensible default for most content sites.

Frequently asked questions

What does noindex, nofollow do?

noindex asks search engines not to show the page in results, and nofollow asks them not to follow the links on it. The tag is <meta name="robots" content="noindex, nofollow">.

What is the difference between the meta tag and X-Robots-Tag?

They carry the same rules. The meta tag sits in the HTML of a page, so it only works for HTML. The X-Robots-Tag is sent as an HTTP header, so it also works for PDFs, images and other files.

Should I use robots.txt or a noindex tag?

robots.txt controls crawling, a noindex tag controls whether a page appears in results. To remove a page from results, use noindex and let it be crawled. Blocking it in robots.txt hides the tag from crawlers.

What does max-snippet do?

It limits how many characters of text a search engine may show as the snippet. max-snippet:-1 means no limit, max-snippet:0 means no text snippet, and a number such as 160 caps it at that many characters.

Can I target one search engine only?

Yes. Change the name from robots to the crawler name, such as googlebot or bingbot. The tool does this when you pick a crawler, and the header version prefixes the crawler name.

Related: Robots.txt generator, Meta tag generator, Canonical tag checker.

Start for free or see all free tools. We.Inc is free to start, with no card needed. AI building is on paid plans from $20 a month.

More in Free Tools

Product

Who It's For

Features

Resources

Company

View Sitemap