NewFirst month of Starter or Pro for $1Your first month of Starter or Pro for just $1, then cancel anytimeSee pricing
Ranken logoRanken.io

ClaudeBot: what it is and how to block it

What is ClaudeBot?

ClaudeBot is Anthropic's crawler for collecting public web content that may be used to train Claude models. Anthropic says it honors robots.txt, including Crawl-delay.

Operator
Anthropic
Used for
Model training
robots.txt token
ClaudeBot
robots.txt
Honors robots.txt

Checked against Anthropic's documentation

robots.txtClaudeBot
User-agent: *
Allow: /
 
User-agent: ClaudeBot
Disallow: /
With this rule
Anthropic model trainingBlocked
Claude search (Claude-SearchBot)Not affected

What is the ClaudeBot user agent string?

Anthropic does not publish a full user agent string. Match the ClaudeBot token instead.

How do I block ClaudeBot in robots.txt?

Add these two lines to the robots.txt file at the root of your domain. Anthropic says its crawler respects robots.txt disallow rules and supports the Crawl-delay directive.

User-agent: ClaudeBot
Disallow: /

To allow it everywhere instead, use:

User-agent: ClaudeBot
Allow: /

What happens if you block ClaudeBot?

Anthropic leaves your future content out of its training data. Claude-SearchBot and Claude-User have their own tokens and are not affected.

Should you block ClaudeBot?

Block it to keep new content out of Anthropic's training data. To stay visible when people search with Claude, keep Claude-SearchBot and Claude-User allowed.

ClaudeBot questions

Why is ClaudeBot crawling my site?

ClaudeBot collects public web pages that Anthropic may use to train Claude models, so it visits sites across the open web. A visit does not mean anything is wrong with your site.

How do I slow ClaudeBot down instead of blocking it?

Anthropic says ClaudeBot supports the Crawl-delay directive. Add a line such as Crawl-delay: 10 under User-agent: ClaudeBot in robots.txt to space out its requests.

How do I check that a visit is really ClaudeBot?

Compare the visitor's IP address with the list Anthropic publishes for its crawlers. A request that only claims to be ClaudeBot in its user agent can come from anyone.

Does your robots.txt block ClaudeBot?The free AI Visibility Checker reads your robots.txt and shows which AI search and training crawlers can reach a page.Check a URL

See the full AI crawler list or read the official Anthropic docs

A blog AI search can read

Every Ranken blog ships a robots.txt that welcomes AI search crawlers, plus server-rendered pages, schema and llms.txt.