GetMentionedByAIClaude-SearchBot and ClaudeBot: the robots.txt difference
Claude-SearchBot supports Claude search relevance while ClaudeBot is associated with model training. Learn what each control changes before blocking Anthropic bots.
Anthropic now documents separate bots for separate uses. Claude-SearchBot navigates the web to improve the relevance and accuracy of search responses. ClaudeBot accesses web content that may contribute to model training. Treating both names as one crawler creates the same policy mistake seen with OpenAI’s search and training bots.
If the goal is to opt out of training without reducing Claude search visibility, block ClaudeBot and leave Claude-SearchBot crawlable.
Which Anthropic bot does what?
| Bot | Published purpose | Reason to keep separate |
|---|---|---|
| Claude-SearchBot | Improves search-result relevance and accuracy | Blocking may reduce visibility and accuracy in Claude search responses |
| ClaudeBot | Collects web content that may contribute to training | Can be blocked as a training-policy choice without automatically blocking search |
The user-agent distinction is a control surface, not an optimization trick. Allowing Claude-SearchBot does not guarantee that Claude will cite a page. It means Anthropic’s search crawler is permitted to read it. The page still needs to be accessible, relevant, and useful for the query.
How to configure Claude crawler access
Anthropic says its bots honor standard robots.txt directives and supports the non-standard Crawl-delay extension where appropriate. Put each user agent in its own group when their policies differ. Apply the file at the top level of every subdomain you need to control.
- Use a ClaudeBot group for the training decision.
- Use a Claude-SearchBot group for the search-indexing decision.
- Avoid IP-only blocking as the primary opt-out; Anthropic warns that IP ranges can change and blocking can prevent the bot from reading robots.txt.
- Inspect WAF and CDN rules if logs still show denied requests after robots.txt allows the crawler.
Test the rule that actually wins
A robots file can contain a broad `User-agent: *` group and a more specific Claude group. The specific group and the longest matching path rule determine the result. Use the AI crawler checker to see the applied rule rather than searching the file for the word “Claude.”
For a wider technical baseline, run the AI crawl checklist as well. It keeps crawler permission, sitemap discovery, llms.txt, and ai.txt as separate observations instead of mixing them into a fictional visibility score.
Primary source
Anthropic explains bot purposes, robots.txt syntax, Crawl-delay, and opt-out behavior in Does Anthropic crawl data from the web?. Check the live document before changing production policy.
Common questions
Is Claude-SearchBot the same as ClaudeBot?
No. Anthropic documents Claude-SearchBot for search relevance and ClaudeBot for web content that may contribute to training.
Does Anthropic respect robots.txt?
Anthropic says its documented bots honor industry-standard robots.txt directives.
Can I slow ClaudeBot instead of blocking it?
Anthropic documents support for the non-standard Crawl-delay extension, though behavior should be verified in your logs.