robots.txt for AI Bots: Which to Allow and Which to Block | How-to Guide
A practical guide to the AI crawlers a small business site sees, what each one is for according to its owner, and robots.txt rules that keep you in AI search answers while opting out of training if you want.
AI companies now send several different crawlers, and each has a different job. Some gather training data. Some build a search index that answers link back to. Some fetch a single page because a user asked. Blocking them all in one line feels safe but can quietly remove a business from AI search answers that send real customers.
How robots.txt works
robots.txt is a plain file at the root of your domain. Each group starts with a User-agent line naming a crawler, followed by Allow or Disallow rules. The format is standardised in RFC 9309. It is a request, not a lock: reputable crawlers follow it, but it does not protect private content, and Google's documentation notes that blocking crawling does not by itself remove a page from the index.
The main AI crawlers and what their owners say they do
- OAI-SearchBot (OpenAI): used to surface sites in ChatGPT search results. OpenAI's crawler documentation says sites that block it will not be shown in ChatGPT search answers, though they may still appear as plain navigational links.
- GPTBot (OpenAI): collects content that may be used to train OpenAI's models. Blocking it is OpenAI's documented way to opt out of training, and it is independent of OAI-SearchBot.
- ChatGPT-User (OpenAI): fetches a page when a person's request needs it. OpenAI notes robots.txt rules may not apply to these user-initiated visits.
- Google-Extended (Google): not a separate crawler but a product token. Disallowing it tells Google not to use your content for training future Gemini models or for grounding in some of its other AI systems. Google Search Central says it does not affect inclusion or ranking in Google Search, and AI Overviews are governed by Googlebot and the normal snippet controls.
- PerplexityBot (Perplexity): indexes pages for Perplexity's answers and, per Perplexity's docs, is not used for model training. Perplexity-User fetches on a user's behalf and generally ignores robots.txt.
- Bingbot (Microsoft): feeds Bing, which grounds Microsoft Copilot. Blocking it removes you from both.
- ClaudeBot and Claude-SearchBot (Anthropic): Anthropic documents separate agents for training and for search, and says they honour robots.txt.
- Applebot and Applebot-Extended (Apple): Applebot powers search features in Spotlight, Siri and Safari; Applebot-Extended lets you opt out of Apple using content to train its foundation models, without affecting search.
A sensible setup for a local business
Most small businesses want to be found and named in answers. A common, balanced setup allows search crawlers and decides separately about training. For example, to stay in ChatGPT search and Perplexity while opting out of OpenAI and Google training:
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlWhether to opt out of training is a business choice. Being in training data can make a model more familiar with your name; opting out keeps your text out of future model builds. Neither choice affects Google Search ranking.
Other places bots get blocked
Check your CDN and security settings as well as robots.txt. Some hosting firewalls and bot-management features block AI crawlers by default, which overrides an open robots.txt. If you use one, look for an AI crawler setting and confirm the search agents you want are allowed.
Test your file
Open yoursite.com/robots.txt in a browser to confirm it loads. Google Search Console has a robots.txt report showing the version Google fetched and any errors. Bing Webmaster Tools offers a robots.txt tester. Re-check after changing hosts or installing a security plugin.
Once the right bots can reach you, they need pages worth reading, which is where SEO God Mode fits on a We.Inc site, suggesting weekly topics from your own services and drafting posts you approve.
Frequently asked questions
If I block GPTBot, will I disappear from ChatGPT search?
Not according to OpenAI. GPTBot is the training crawler; ChatGPT search uses OAI-SearchBot. Allowing one while blocking the other is fine.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended controls use of content for Gemini training and does not affect Google Search, including AI Overviews, which use Googlebot.
Is robots.txt enough to keep content private?
No. It is a request that well-behaved crawlers follow. Anything truly private needs a login or password protection, and noindex is the tool for keeping a page out of search results.
More in Guides
- Roofing SEO: A Practical Checklist for Roofers (2026) | How-to Guide
- Sales Tax for Online Stores: A Plain-English US Guide
- School District Website Accessibility: A Title II Plan for the District Site and Every School Site | How-to Guide
- Scroll Animations: Four Effects, Three Ways
- Seasonal Content Planning for a Small Business | How-to Guide
- Section 508 vs ADA Title II vs WCAG: Which Applies to Your Website? | How-to Guide
- Self-Hosted AI Agent: What You Run, and What You Must Lock Down
- How to Send a Website Form to Email (5 Methods With Code)
Website builders for the businesses in this article
Related guides and tools
Start building
Product
Who It's For
- For Agencies
- For Freelancers
- For Hosting Providers
- For SaaS Platforms
- For Startups
- For Financial Services
- For Healthcare
Features
Resources
- Blog
- Help Center
- Free Business Tools
- Free UI Components
- Website Templates
- Template Collections
- Website Examples
- Guides
- Comparisons
- Alternatives
- Best Lists
- Reviews
- Pricing Guides
- Use Cases
- Glossary & Guides
- Integrations
- Developers
- Benchmark
- Windows App