Two lines, User-agent: * followed by Disallow: /, ask every polite crawler to stop fetching any page on that host. They do not remove pages from search: Google says a blocked address can still be listed without a description when other sites link to it. To keep pages out, use noindex or a password.
| Method | Stops crawling | Keeps the page out of results |
|---|---|---|
| robots.txt Disallow: / | Yes, for crawlers that obey it | No, the bare address can still appear |
| noindex meta tag or X-Robots-Tag header | No, the page must stay crawlable | Yes, once Google recrawls it |
| Password or login | Yes, crawlers get no content | Yes |
| noindex line inside robots.txt | No | No, Google does not support it |
Updated
robots.txt is a plain text file at the root of your domain that tells crawlers, the programs search engines send to read pages, which paths they may fetch. A disallow all file is common on staging copies and new sites, and it is also the line most often left behind by accident at launch. Test your own rules in the matcher below.
Your robots.txt
User-agent: *
Disallow: /
Allow: /public/
User-agent: Bingbot
Crawl-delay: 5
Disallow: /cart/Crawler name: Googlebot
Path to test: /public/menu.pdf
Allowed: /public/menu.pdf may be crawled.
Group used: User-agent: * (no group names this crawler).
Deciding line 3: Allow: /public/
Runs in your browser on the text you paste, following RFC 9309. Nothing is fetched or uploaded.
Each file below goes at the root of one host, for example https://staging.example.com/robots.txt.
The classic staging file. Remove it before launch.
User-agent: *
Disallow: /An empty Disallow value blocks nothing. Add your sitemap address so crawlers find every page.
User-agent: *
Disallow:
Sitemap: https://www.example.com/sitemap.xmlThe longer Allow path beats the one-character Disallow path.
User-agent: *
Disallow: /
Allow: /press/Leave the page crawlable and add this tag to its head. The response header form works for PDFs and images.
<meta name="robots" content="noindex">
<!-- or as an HTTP response header -->
X-Robots-Tag: noindexFirst the crawler picks a group. It looks for a User-agent line that names it, and only when none matches does it fall back to the group headed User-agent: *. Then, within that group, the rule with the longest matching path wins. When an allow rule and a disallow rule are equally long, RFC 9309 says the allow rule should be used.
That is why Disallow: / plus Allow: /public/ still opens the public folder: the allow path is longer, so it is more specific.
A development copy at a staging address can end up in search if anyone links to it. A login or HTTP authentication stops both crawling and indexing. If you can only use robots.txt, pair it with care at launch: delete the block, then request indexing of the home page in Search Console.
If staging pages already appear in results, remove the robots.txt block for a while so Google can crawl them and see a noindex tag, or protect them with a password.
No. It stops Googlebot from reading pages, but Google's documentation says a disallowed URL may still be indexed and shown without a snippet. To drop pages from results, let Google crawl them and serve noindex instead.
Google's block-indexing page states that a noindex rule in the robots.txt file is not supported. Put noindex in a robots meta tag in the page head, or in an X-Robots-Tag response header for files like PDFs.
No. Google's robots.txt specification lists user-agent, allow, disallow and sitemap as the fields it reads and names crawl-delay as unsupported. Bing's webmaster blog says Bingbot does honour Crawl-delay, with a window from one to thirty seconds.
RFC 9309 says a crawler must assume complete disallow when the file is unreachable because of a server error. Google stops crawling for the first 12 hours and then falls back to its last good copy, so a broken file can pause crawling.
Google generally caches the file for up to 24 hours, and RFC 9309 says crawlers should not use a cached copy longer than that. Expect up to a day before a removed block is noticed.
Related: robots.txt tester, robots.txt generator, Sitemap and robots.txt examples, WordPress robots.txt, Google Search Console guide.
Start for free or see all free tools. We.Inc is free to start, with no card needed. AI building is on paid plans from $20 a month.
We.Inc is an AI-powered website builder you can resell under your own brand. Launch a branded client dashboard, bill on Stripe Connect, and deliver AI-generated websites in minutes. White-label plans start at $99 a month for 25 client sites, with a 7-day free trial and no per-site fees.