CCBot: what it is and how to block it
What is CCBot?
CCBot is the crawler behind Common Crawl, a free and open archive of the web. Many AI models have been trained on Common Crawl data, so blocking CCBot keeps new pages out of a dataset that AI labs use.
- Operator
- Common Crawl
- Used for
- Model training
- robots.txt token
CCBot- robots.txt
- Opt out in robots.txt
- IP ranges
- Published list
Checked against Common Crawl's documentation
What is the CCBot user agent string?
CCBot/2.0 (https://commoncrawl.org/faq/)As published by Common Crawl.
How do I block CCBot in robots.txt?
Add these two lines to the robots.txt file at the root of your domain. Common Crawl documents a robots.txt rule to block CCBot and also runs a separate opt-out registry.
User-agent: CCBot
Disallow: /To allow it everywhere instead, use:
User-agent: CCBot
Allow: /What happens if you block CCBot?
New pages stay out of future Common Crawl archives. It has no effect on Google, Bing or AI search answers.
Should you block CCBot?
Block it if you do not want your pages in an open dataset that AI models train on. Search visibility is not affected.
Other AI crawlers
- GPTBotOpenAI, model training
- OAI-SearchBotOpenAI, AI search
- ChatGPT-UserOpenAI, user-triggered
- ClaudeBotAnthropic, model training
- PerplexityBotPerplexity, AI search
- Perplexity-UserPerplexity, user-triggered
See the full AI crawler list or read the official Common Crawl docs
A blog AI search can read
Every Ranken blog ships a robots.txt that welcomes AI search crawlers, plus server-rendered pages, schema and llms.txt.