Guide · Updated 2026

Robots.txt for AI Search: The Complete 2026 Guide to AI Crawlers

Every major AI company now runs dedicated web crawlers to train models and retrieve live search results. Your robots.txt file determines which AI models can read your content, which are blocked, and whether you appear in AI search results at all. This guide covers how each AI crawler works and how to configure them.

The two types of AI crawlers: search vs training

Understanding the distinction between search crawlers and training crawlers is critical. Search crawlers (ChatGPT User, PerplexityBot) fetch web pages in real-time when users ask questions. Blocking them means your content cannot appear in AI search answers. Training crawlers (GPTBot, ClaudeBot, Google-Extended) crawl the web to build future models. You can allow search crawlers while blocking training crawlers if you want AI search traffic without contributing to model training data.

Major AI crawler User-agent tokens in 2026

OpenAI: GPTBot (training) and ChatGPT-User (real-time search). Anthropic: ClaudeBot (training) and Claude-Web (search). Perplexity: PerplexityBot (search and indexing). Google: Googlebot (all Google search, including AI Overviews) and Google-Extended (Gemini model training). Mistral: MistralBot. Cohere: cohere-ai. Apple: Applebot-Extended.

Recommended robots.txt configuration for maximum AI visibility

If your goal is maximum traffic and citations from AI search engines: allow all search crawlers explicitly. At minimum: User-agent: GPTBot / Allow: / — User-agent: PerplexityBot / Allow: / — User-agent: ClaudeBot / Allow: / — User-agent: Googlebot / Allow: /. Always block private and authenticated paths: Disallow: /api/, Disallow: /dashboard/, Disallow: /admin/.

Common robots.txt mistakes that kill AI search visibility

1. Blocking GPTBot thinking it only trains models — it also feeds ChatGPT Search retrieval in many configurations. 2. Forgetting that Google AI Overviews uses Googlebot, not Google-Extended — blocking Google-Extended does not stop Google AI Overviews from citing you. 3. Missing the Sitemap directive in robots.txt — AI crawlers use your sitemap to discover new content quickly. 4. Disallowing CSS/JS assets that AI headless browsers need to render page content.

Ready to put this into practice?

Audit your robots.txt with OptiAISEO →

No account required

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?+

It stops GPTBot from crawling your site for training and reduces your visibility in ChatGPT Search. If your content is already in the model's training weights, blocking GPTBot will not erase that historical knowledge immediately, but it prevents updates.

Can I block AI scrapers without hurting Google SEO?+

Yes. Google uses Googlebot for search ranking and Google-Extended for Gemini training. You can disallow Google-Extended in robots.txt without any impact on your traditional Google search rankings or Googlebot crawling.

Where should the Sitemap directive be in robots.txt?+

At the end of your robots.txt file: Sitemap: https://yourdomain.com/sitemap.xml. Use the full absolute URL including https://.