NewFirst month of Starter or Pro for $1Your first month of Starter or Pro for just $1, then cancel anytimeSee pricing
Ranken logoRanken.io

CCBot: what it is and how to block it

What is CCBot?

CCBot is the crawler behind Common Crawl, a free and open archive of the web. Many AI models have been trained on Common Crawl data, so blocking CCBot keeps new pages out of a dataset that AI labs use.

Operator
Common Crawl
Used for
Model training
robots.txt token
CCBot
robots.txt
Opt out in robots.txt

Checked against Common Crawl's documentation

robots.txtCCBot
User-agent: *
Allow: /
 
User-agent: CCBot
Disallow: /
With this rule
Future Common Crawl archivesBlocked
Google, Bing and AI searchNot affected

What is the CCBot user agent string?

CCBot/2.0 (https://commoncrawl.org/faq/)

As published by Common Crawl.

How do I block CCBot in robots.txt?

Add these two lines to the robots.txt file at the root of your domain. Common Crawl documents a robots.txt rule to block CCBot and also runs a separate opt-out registry.

User-agent: CCBot
Disallow: /

To allow it everywhere instead, use:

User-agent: CCBot
Allow: /

What happens if you block CCBot?

New pages stay out of future Common Crawl archives. It has no effect on Google, Bing or AI search answers.

Should you block CCBot?

Block it if you do not want your pages in an open dataset that AI models train on. Search visibility is not affected.

Which AI crawlers does your site block?The free AI Visibility Checker reads your robots.txt and shows which AI search and training crawlers can reach a page.Check a URL

See the full AI crawler list or read the official Common Crawl docs

A blog AI search can read

Every Ranken blog ships a robots.txt that welcomes AI search crawlers, plus server-rendered pages, schema and llms.txt.