AI crawlers list and the robots.txt rules for each
AI crawlers do three different jobs. Search crawlers decide whether AI answers cite you, user agents fetch a page when someone asks, and training crawlers collect content for models. Block the training ones if you want, and keep the search ones allowed.
10 crawlers, checked against each operator's documentation
AI search crawlers
They decide whether an AI search engine can cite and link your pages. Blocking them removes you from those answers.
- OAI-SearchBotOpenAIAllow it if you want ChatGPT to cite and link your pages. Blocking OAI-SearchBot is the main way a site drops out of ChatGPT search answers.Opt out in robots.txtDetails
- PerplexityBotPerplexityAllow it if you want Perplexity to cite and link your pages. It is a search crawler, not a training crawler.Opt out in robots.txtDetails
User-triggered agents
They fetch a page only when a person asks for it. Operators say robots.txt may not apply to them.
- ChatGPT-UserOpenAILeave it allowed. It only visits when someone asks ChatGPT about your page, and blocking it means ChatGPT cannot read the page for that person.robots.txt may not applyDetails
- Perplexity-UserPerplexityLeave it allowed. Its visits come from real people looking for an answer, and robots.txt is not a reliable way to stop it anyway.Generally ignores robots.txtDetails
Training crawlers and tokens
They control whether your content trains AI models. Blocking them does not remove you from search.
- GPTBotOpenAIBlock it if you do not want your content used to train OpenAI models. It has no effect on ChatGPT search, so a common setup is to block GPTBot and allow OAI-SearchBot.Opt out in robots.txtDetails
- ClaudeBotAnthropicBlock it to keep new content out of Anthropic's training data. To stay visible when people search with Claude, keep Claude-SearchBot and Claude-User allowed.Honors robots.txtDetails
- Google-ExtendedGoogleBlock it if you do not want your content used for Gemini training and grounding. Your place in Google Search stays the same.Control tokenDetails
- Applebot-ExtendedAppleBlock it to opt out of Apple model training while staying in Siri, Spotlight and Safari search. Keep Applebot itself allowed.Control tokenDetails
- Meta-ExternalAgentMetaBlock it if you do not want Meta to use your content for AI training. Link previews on Facebook and Instagram use a different agent.Opt out in robots.txtDetails
- CCBotCommon CrawlBlock it if you do not want your pages in an open dataset that AI models train on. Search visibility is not affected.Opt out in robots.txtDetails
Which AI crawlers respect robots.txt?
Most automatic crawlers follow robots.txt. Agents that fetch a page because a person asked for it may not, and two entries are tokens rather than crawlers.
| Crawler | Operator | Used for | robots.txt |
|---|---|---|---|
| GPTBot | OpenAI | Model training | Opt out in robots.txt |
| OAI-SearchBot | OpenAI | AI search | Opt out in robots.txt |
| ChatGPT-User | OpenAI | User-triggered fetch | robots.txt may not apply |
| ClaudeBot | Anthropic | Model training | Honors robots.txt |
| PerplexityBot | Perplexity | AI search | Opt out in robots.txt |
| Perplexity-User | Perplexity | User-triggered fetch | Generally ignores robots.txt |
| Google-Extended | Model training | Control token | |
| Applebot-Extended | Apple | Model training | Control token |
| Meta-ExternalAgent | Meta | Model training | Opt out in robots.txt |
| CCBot | Common Crawl | Model training | Opt out in robots.txt |
What should robots.txt say for AI crawlers?
Allow AI search crawlers and block AI training: name the search crawlers with Allow and the training ones with Disallow. Paste this into the robots.txt at the root of your domain, below your existing rules.
# Allow AI search and answer crawlers
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Opt out of AI model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: CCBot
Disallow: /Then run any page through the free AI Visibility Checker to confirm which crawlers can reach it. Want an llms.txt file too? Use the llms.txt generator.
A blog AI search can read
Every Ranken blog ships a robots.txt that welcomes AI search crawlers, plus server-rendered pages, schema and llms.txt.