Build a correct robots.txt file in a minute: choose a default for all crawlers, list folders to keep out, add your sitemap, and decide whether AI crawlers may read your content. Copy the result or download it as a file.
A robots.txt file is a list of groups. Each group starts with one or more User-agent lines naming a crawler (or * for everyone), followed by Allow and Disallow lines with URL paths. A crawler uses the most specific group that names it, and within a group the longest matching path wins.
'Disallow: /' blocks everything. An empty 'Disallow:' allows everything. 'Disallow: /admin/' blocks every address starting with /admin/. Paths are case sensitive.
Allow everything, block any admin, cart, checkout or account folders your platform uses, and list your sitemap. That is all most sites need. Do not use robots.txt to hide thank-you pages or duplicate pages from Google; a noindex tag does that job properly.
Every site published with We.Inc already serves a robots.txt and a sitemap automatically, so you only need this generator for sites hosted elsewhere.
It tells well-behaved crawlers which parts of your site they may request. It is a request, not a lock: it does not protect private content, and bad bots ignore it.
Not strictly. Without one, crawlers assume they may crawl everything. Having one is still useful to point to your sitemap and to keep crawlers out of pages like carts and admin screens.
It is a business choice. Blocking GPTBot, ClaudeBot, Google-Extended and similar crawlers asks them not to use your content for training, but may also mean AI assistants know less about your business. Many small businesses leave them allowed so AI answers can recommend them.
Google usually re-reads robots.txt within a day. You can ask for a refresh in the robots.txt report in Google Search Console.
Related: Schema Markup Generator, Meta Tag Generator, Website Checker, DNS Lookup.
Start for free or see how the We.Inc AI website builder works. The free plan gives you up to 3 template sites on a we.inc address, with no AI and no custom domain. AI building starts at $20 a month.
We.Inc is an AI-powered website builder you can resell under your own brand. Launch a branded client dashboard, bill on Stripe Connect, and deliver AI-generated websites in minutes. White-label plans start at $99 a month for 25 client sites, with a 7-day free trial and no per-site fees.