NewFirst month of Starter or Pro for $1Your first month of Starter or Pro for just $1, then cancel anytimeSee pricing
Ranken logoRanken.io

AI crawlers list and the robots.txt rules for each

AI crawlers do three different jobs. Search crawlers decide whether AI answers cite you, user agents fetch a page when someone asks, and training crawlers collect content for models. Block the training ones if you want, and keep the search ones allowed.

10 crawlers, checked against each operator's documentation

AI search crawlers

They decide whether an AI search engine can cite and link your pages. Blocking them removes you from those answers.

User-triggered agents

They fetch a page only when a person asks for it. Operators say robots.txt may not apply to them.

Training crawlers and tokens

They control whether your content trains AI models. Blocking them does not remove you from search.

Which AI crawlers respect robots.txt?

Most automatic crawlers follow robots.txt. Agents that fetch a page because a person asked for it may not, and two entries are tokens rather than crawlers.

CrawlerOperatorUsed forrobots.txt
GPTBotOpenAIModel trainingOpt out in robots.txt
OAI-SearchBotOpenAIAI searchOpt out in robots.txt
ChatGPT-UserOpenAIUser-triggered fetchrobots.txt may not apply
ClaudeBotAnthropicModel trainingHonors robots.txt
PerplexityBotPerplexityAI searchOpt out in robots.txt
Perplexity-UserPerplexityUser-triggered fetchGenerally ignores robots.txt
Google-ExtendedGoogleModel trainingControl token
Applebot-ExtendedAppleModel trainingControl token
Meta-ExternalAgentMetaModel trainingOpt out in robots.txt
CCBotCommon CrawlModel trainingOpt out in robots.txt

What should robots.txt say for AI crawlers?

Allow AI search crawlers and block AI training: name the search crawlers with Allow and the training ones with Disallow. Paste this into the robots.txt at the root of your domain, below your existing rules.

# Allow AI search and answer crawlers
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Opt out of AI model training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: CCBot
Disallow: /

Then run any page through the free AI Visibility Checker to confirm which crawlers can reach it. Want an llms.txt file too? Use the llms.txt generator.

A blog AI search can read

Every Ranken blog ships a robots.txt that welcomes AI search crawlers, plus server-rendered pages, schema and llms.txt.