Paste the contents of a robots.txt file, type the page path you want to test and choose a crawler, and the tester tells you whether that crawler is allowed or blocked and which rule decided it. It also lists syntax errors, unknown directives and the sitemap lines it finds. It works only on the text you paste and never fetches your live file.
A robots.txt file is a list of groups. Each group begins with one or more User-agent lines that name the crawlers it addresses, followed by Allow and Disallow lines that give path patterns. A crawler picks the group that names it most specifically, ignores every other group and applies the rules inside it.
Rules match the beginning of the path, including the query string. Disallow: /shop blocks /shop, /shop/ and /shopping-guide alike, because all of them start with /shop. If you mean only the folder, write /shop/ with the closing slash.
The most expensive mistake is Disallow: / left in place after a launch. It is common on staging sites and survives the move to the live domain, and it asks every compliant crawler to stay away from the whole site. The tester flags a rule that blocks everything so you can confirm it was intended.
The next group of mistakes are typing errors. A misspelt directive such as Disalow is ignored without a warning, which means the page you meant to protect stays open. A rule that does not begin with a slash, or a relative sitemap address, is also ignored. Checking the file after every edit catches these before a crawler does.
It collects every Allow and Disallow rule in the matching group whose pattern fits the start of the path, then picks the rule with the longest pattern. If an Allow and a Disallow match with the same length, Allow wins. If no rule matches, the path is allowed. For example, with Disallow: /admin/ and Allow: /admin/help, the path /admin/help/page is allowed because the Allow rule is longer.
An asterisk matches any run of characters, and a dollar sign at the end of a pattern means the path must end there. So Disallow: /*.pdf$ blocks addresses that end in .pdf, but not /file.pdf?download=1, because that address does not end with .pdf.
First a group that names your crawler exactly, for example Googlebot. Image and video crawlers such as Googlebot-Image fall back to Googlebot when they have no group of their own. If there is none, the group for * is used, and if that is missing too, everything is allowed.
No. It reads only what you paste, so what it shows is only as current as your copy. Fetch the live file yourself and paste it to test what search engines will see.
It follows the same published matching rules, the longest match and the * and $ wildcards, but it is an independent tool and does not cover every edge case such as unusual encodings. Check important changes in Google Search Console as well.
Related: Robots.txt generator, Sitemap checker, Canonical tag checker.
Start for free or see all free tools. We.Inc is free to start, with no card needed. AI building is on paid plans from $20 a month.
We.Inc is an AI-powered website builder you can resell under your own brand. Launch a branded client dashboard, bill on Stripe Connect, and deliver AI-generated websites in minutes. White-label plans start at $99 a month for 25 client sites, with a 7-day free trial and no per-site fees.