A sitemap.xml lists the URLs you want crawled, each in a url element with a loc and an optional lastmod, up to 50,000 URLs or 50 MB per file. A robots.txt at the site root tells crawlers which paths to skip with User-agent, Disallow and Allow lines, and can point to the sitemap.
| Item | Rule |
|---|---|
| URLs per sitemap | 50,000 maximum |
| Sitemap file size | 50 MB uncompressed maximum |
| URL format in sitemaps | Fully qualified, for example https://example.com/page |
| robots.txt location | Root of the host, for example https://example.com/robots.txt |
| robots.txt size read by Google | 500 KiB |
| Fields Google supports in robots.txt | user-agent, allow, disallow, sitemap |
Updated
Generators spit out a file, but they rarely explain why each line is there or what happens when two rules disagree. Every example below is annotated, and the builder turns a plain list of your pages into a valid sitemap with the special characters escaped for you.
Paste one full URL per line and the builder writes a valid urlset with special characters escaped.
The xml declaration, the urlset with the protocol namespace, and one url per page. lastmod uses the W3C date format.
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/</loc>
<lastmod>2026-10-01</lastmod>
</url>
<url>
<loc>https://example.com/services/roof-repair</loc>
<lastmod>2026-09-18</lastmod>
</url>
<url>
<loc>https://example.com/search?type=metal&area=north</loc>
</url>
</urlset>Points to child sitemaps. Each child follows the same 50,000 URL and 50 MB limit.
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://example.com/sitemaps/pages.xml</loc>
<lastmod>2026-10-01</lastmod>
</sitemap>
<sitemap>
<loc>https://example.com/sitemaps/products-1.xml</loc>
</sitemap>
</sitemapindex>An empty Disallow means nothing is blocked. This is the right file for most small business sites.
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xmlBlocks URL patterns that create endless duplicates while keeping product and category pages open.
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account/
Disallow: /*?sort=
Disallow: /*?*filter=
Allow: /account/help
Sitemap: https://shop.example.com/sitemap_index.xmlStops compliant crawlers on a test copy. Add a password too, since this file only asks politely.
User-agent: *
Disallow: /The dollar sign means the URL ends there, so /guide.pdf is blocked but /guide.pdf.html is not.
User-agent: Googlebot
Disallow: /*.pdf$
Disallow: /tmp/
User-agent: *
Disallow: /tmp/| Rules | URL | Result | Why |
|---|---|---|---|
| Allow: /p and Disallow: / | /page | Allowed | /p is the longer match |
| Allow: /folder and Disallow: /folder | /folder/page | Allowed | Equal length, least restrictive wins |
| Disallow: /*.php$ | /index.php?x=1 | Allowed | The URL does not end in .php |
| Disallow: /fish | /Fish.html | Allowed | Paths are case-sensitive |
The sitemap is an invitation: here are the pages worth crawling and when they last changed. It helps most on new sites, large sites and pages that few internal links reach.
robots.txt is a set of crawl instructions, not a security tool. Anyone can read it, and badly behaved bots ignore it, so never rely on it to hide private areas. Protect those with a login.
A crawler picks the group whose User-agent line matches it most specifically and ignores the rest. Googlebot with its own group will not read the rules under User-agent: *.
Paths are case-sensitive and match from the start. An asterisk stands for any run of characters and a dollar sign anchors the end of the URL, so Disallow: /*.pdf$ blocks every PDF.
Anywhere on the same host works, but the root is the usual place because a sitemap generally covers only URLs at or below its own folder under the sitemaps.org protocol. Tell search engines where it is with a Sitemap line in robots.txt or by submitting it in Search Console.
Split them across several sitemap files and list those files in a sitemap index. The index uses sitemapindex and sitemap elements instead of urlset and url, and it can reference up to 50,000 sitemaps.
No. Google reads only user-agent, allow, disallow and sitemap. Other crawlers may honour Crawl-delay, but Googlebot ignores it, and noindex has to go in a meta tag or X-Robots-Tag header.
Google picks the most specific rule, meaning the one with the longest matching path. If two matching rules are equally specific, it uses the less restrictive one, which is Allow.
Yes. A sitemap is XML, so an ampersand must be written as & and the other reserved characters as their entities. The file must also be UTF-8 encoded.
Related: XML sitemap generator, robots.txt generator, robots.txt tester, Sitemap checker, Next.js SEO checklist.
Start for free or see all free tools. We.Inc is free to start, with no card needed. AI building is on paid plans from $20 a month.
We.Inc is an AI-powered website builder you can resell under your own brand. Launch a branded client dashboard, bill on Stripe Connect, and deliver AI-generated websites in minutes. White-label plans start at $99 a month for 25 client sites, with a 7-day free trial and no per-site fees.