ClaudeBot: what it is and how to block it
What is ClaudeBot?
ClaudeBot is Anthropic's crawler for collecting public web content that may be used to train Claude models. Anthropic says it honors robots.txt, including Crawl-delay.
- Operator
- Anthropic
- Used for
- Model training
- robots.txt token
ClaudeBot- robots.txt
- Honors robots.txt
- IP ranges
- Published list
Checked against Anthropic's documentation
What is the ClaudeBot user agent string?
Anthropic does not publish a full user agent string. Match the ClaudeBot token instead.
How do I block ClaudeBot in robots.txt?
Add these two lines to the robots.txt file at the root of your domain. Anthropic says its crawler respects robots.txt disallow rules and supports the Crawl-delay directive.
User-agent: ClaudeBot
Disallow: /To allow it everywhere instead, use:
User-agent: ClaudeBot
Allow: /What happens if you block ClaudeBot?
Anthropic leaves your future content out of its training data. Claude-SearchBot and Claude-User have their own tokens and are not affected.
Should you block ClaudeBot?
Block it to keep new content out of Anthropic's training data. To stay visible when people search with Claude, keep Claude-SearchBot and Claude-User allowed.
ClaudeBot questions
Why is ClaudeBot crawling my site?
ClaudeBot collects public web pages that Anthropic may use to train Claude models, so it visits sites across the open web. A visit does not mean anything is wrong with your site.
How do I slow ClaudeBot down instead of blocking it?
Anthropic says ClaudeBot supports the Crawl-delay directive. Add a line such as Crawl-delay: 10 under User-agent: ClaudeBot in robots.txt to space out its requests.
How do I check that a visit is really ClaudeBot?
Compare the visitor's IP address with the list Anthropic publishes for its crawlers. A request that only claims to be ClaudeBot in its user agent can come from anyone.
Other AI crawlers
- GPTBotOpenAI, model training
- OAI-SearchBotOpenAI, AI search
- ChatGPT-UserOpenAI, user-triggered
- PerplexityBotPerplexity, AI search
- Perplexity-UserPerplexity, user-triggered
- Google-ExtendedGoogle, model training
See the full AI crawler list or read the official Anthropic docs
A blog AI search can read
Every Ranken blog ships a robots.txt that welcomes AI search crawlers, plus server-rendered pages, schema and llms.txt.