WordPress has no robots.txt file on disk by default. When someone requests /robots.txt, core builds one on the fly: User-agent: *, Disallow: /wp-admin/, Allow: /wp-admin/admin-ajax.php, plus a Sitemap line on public sites since version 5.5. Uploading a real robots.txt to the site root replaces that virtual version.
| Line | Where it comes from | Why |
|---|---|---|
| User-agent: * | do_robots() | The rules apply to every crawler |
| Disallow: /wp-admin/ | do_robots(), path of admin_url() | Keeps crawlers out of the dashboard |
| Allow: /wp-admin/admin-ajax.php | do_robots() | Themes and plugins call this file from public pages |
| Sitemap: .../wp-sitemap.xml | WP_Sitemaps::add_robots() (5.5+) | Points crawlers at the core sitemap index; only on public sites |
Updated
Because the default file is generated by code, you will not find it with an FTP client or in your hosting file manager, which confuses many site owners. You can change it three ways: upload a real file, let an SEO plugin edit it, or add a small filter in code. Build a safe version with the generator below.
Sitemap address
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Sitemap: https://www.example.com/wp-sitemap.xml
Built in your browser. Nothing is uploaded or fetched.
Add this to a small custom plugin or your child theme's functions.php. It only runs when no real robots.txt exists in the root.
The second argument tells you whether the site is public, so a staging copy can be left alone.
add_filter( 'robots_txt', function ( $output, $public ) {
if ( ! $public ) {
return $output;
}
$output .= "Disallow: /?s=\n";
$output .= "Disallow: /search/\n";
return $output;
}, 10, 2 );Generated by core on a site installed at the domain root, with the core sitemap enabled.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/wp-sitemap.xmlWordPress routes /robots.txt to the do_robots function. It starts the output with User-agent: *, adds a Disallow line for the admin folder and an Allow line for admin-ajax.php, then runs the result through the robots_txt filter. The WP_Sitemaps class uses that same filter to append the Sitemap line when the site is public.
Plugins hook the same filter, which is why an SEO plugin can show a robots.txt editor without creating a file.
Internal search result pages (addresses with ?s=) can create endless thin pages, so many sites block them. WooCommerce stores often block the cart, checkout and account pages, which never need to appear in search.
Keep the list short. Each extra line is a chance to block something important by mistake, and robots.txt is not a privacy tool.
Usually nowhere. The do_robots function writes the response each time /robots.txt is requested. Only if you or a plugin uploaded a real robots.txt to the root folder is there a file to edit.
Not since WordPress 5.3. The Reading Settings article says the option now adds a robots meta tag with noindex,nofollow to the head of each page. Until 5.2 it made robots.txt send Disallow: /.
Either upload a plain text file named robots.txt to the site root, or hook the robots_txt filter in a small plugin or your child theme. The filter receives the generated text and a flag saying whether the site is public.
The Reading Settings article notes the built-in behaviour only works when WordPress is installed in the site root and no robots.txt exists. A physical file, a host-level rule or a caching layer can serve something else.
The default already does, and that is fine: the dashboard has nothing to rank. Keep the Allow line for admin-ajax.php so public features that call it are not blocked.
Related: robots.txt disallow all, robots.txt tester, robots.txt generator, Sitemap and robots.txt examples, WordPress .htaccess.
Start for free or see all free tools. We.Inc is free to start, with no card needed. AI building is on paid plans from $20 a month.
We.Inc is an AI-powered website builder you can resell under your own brand. Launch a branded client dashboard, bill on Stripe Connect, and deliver AI-generated websites in minutes. White-label plans start at $99 a month for 25 client sites, with a 7-day free trial and no per-site fees.