Build a correct robots.txt file in a minute: choose a default for all crawlers, list folders to keep out, add your sitemap, and decide whether AI crawlers may read your content. Copy the result or download it as a file.
A robots.txt file is a list of groups. Each group starts with one or more User-agent lines naming a crawler (or * for everyone), followed by Allow and Disallow lines with URL paths. A crawler uses the most specific group that names it, and within a group the longest matching path wins.
'Disallow: /' blocks everything. An empty 'Disallow:' allows everything. 'Disallow: /admin/' blocks every address starting with /admin/. Paths are case sensitive.
Allow everything, block any admin, cart, checkout or account folders your platform uses, and list your sitemap. That is all most sites need. Do not use robots.txt to hide thank-you pages or duplicate pages from Google; a noindex tag does that job properly.
Every site published with We.Inc already serves a robots.txt and a sitemap automatically, so you only need this generator for sites hosted elsewhere.
A small mistake in robots.txt can block a whole section of a site, so test the addresses that matter before you upload: your homepage, a product or service page, a blog post, and anything you meant to block. Enter each one in the tester with the crawler you care about, usually Googlebot, and check that the result and the deciding rule are what you expect.
Wildcards are where most surprises come from. 'Disallow: /*?sort=' blocks any address containing ?sort=, and 'Disallow: /*.pdf$' blocks addresses that end in .pdf but not /file.pdf?download=1. Also remember that a crawler named in its own group, such as Googlebot-Image, ignores the * group completely, so rules you wrote for everyone do not apply to it.
It tells well-behaved crawlers which parts of your site they may request. It is a request, not a lock: it does not protect private content, and bad bots ignore it.
Not strictly. Without one, crawlers assume they may crawl everything. Having one is still useful to point to your sitemap and to keep crawlers out of pages like carts and admin screens.
It is a business choice. Blocking GPTBot, ClaudeBot, Google-Extended and similar crawlers asks them not to use your content for training, but may also mean AI assistants know less about your business. Many small businesses leave them allowed so AI answers can recommend them.
Google usually re-reads robots.txt within a day. You can ask for a refresh in the robots.txt report in Google Search Console.
Use the tester on this page. Paste your robots.txt (or use the one you just generated), enter the address and the crawler name, such as Googlebot, and it shows Allowed or Blocked with the exact rule that decided. It applies Google's matching rules in your browser and does not fetch your live file, so paste the current version from yourdomain.com/robots.txt.
For Google, the rule with the longer path wins, because it is more specific. If both paths are the same length, Allow wins. Other crawlers may resolve conflicts differently, so avoid relying on ties.
Related: Schema Markup Generator, Meta Tag Generator, Website Checker, DNS Lookup.
Start for free or see how the We.Inc AI website builder works. The free plan gives you up to 3 template sites on a we.inc address, with no AI and no custom domain. AI building starts at $20 a month.
We.Inc is an AI-powered website builder you can resell under your own brand. Launch a branded client dashboard, bill on Stripe Connect, and deliver AI-generated websites in minutes. White-label plans start at $99 a month for 25 client sites, with a 7-day free trial and no per-site fees.