What is Claude-SearchBot? AI crawler guide
Claude-SearchBot by Anthropic: Anthropic search crawler that indexes content to improve Claude search result relevance and accuracy. Check its reported user-agent, robots.txt behavior, source, and verification guidance.
Anthropic search crawler that indexes content to improve Claude search result relevance and accuracy.
What is Claude-SearchBot?
Claude-SearchBot is a web crawler operated by Anthropic. It indexes content to improve the relevance and accuracy of search results within Claude. The bot identifies itself with the user-agent token Claude-SearchBot and respects the Robots Exclusion Protocol. It is distinct from other Anthropic crawlers and is used solely for search indexing purposes. Site owners can control its access through standard robots.txt directives. It does not collect data for training language models, only for enhancing search functionality within Claude.
What it's for
If you run a website, allowing Claude-SearchBot can help your content appear in Claude search results, potentially increasing visibility to users who rely on Claude for information. Blocking it may reduce your site's presence and accuracy in those results, which could affect traffic from Claude users.
Allowing Claude-SearchBot creates the possibility of retrieval. To find out whether Claude actually cites or recommends you, track your Claude mentions over time.
How to handle Claude-SearchBot
To manage Claude-SearchBot, add a rule in your robots.txt file targeting the user-agent Claude-SearchBot. You can disallow specific paths or the entire site. The bot honors robots.txt directives, so this is the primary method to control its crawling.
robots.txt rule
User-agent: Claude-SearchBot Disallow: /
Blocking cost
Blocking Claude-SearchBot can reduce your site's visibility and accuracy in Claude search results, potentially limiting traffic from users who rely on Claude for information.
Examples
- A news website allows Claude-SearchBot to crawl its articles, leading to those articles appearing in Claude search results when users ask about current events.
- An e-commerce site blocks Claude-SearchBot via robots.txt, and its product pages no longer show up in Claude search results, reducing referral traffic.
- A blog partially allows Claude-SearchBot by disallowing certain directories, resulting in only selected posts being indexed and surfaced in Claude searches.
Related bots
- Kagibot: Also tracked as a search crawler.
- Amzn-SearchBot: Also tracked as a search crawler.
- Kimi-SearchBot: Also tracked as a search crawler.
- OAI-SearchBot: Also tracked as a search crawler.
- Meta-WebIndexer: Also tracked as a search crawler.
- ExaSearchBot: Also tracked as a search crawler.
- MistralAI-Index: Also tracked as a search crawler.
- PerplexityBot: Also tracked as a search crawler.
- xAI-SearchBot: Also tracked as a search crawler.
- Anthropic: Claude-SearchBot connects this operator term to its crawler behavior.
- Claude: Claude-SearchBot gives crawler context for Claude.
Frequently Asked Questions
What does Claude-SearchBot do?
Claude-SearchBot is a crawler from Anthropic that indexes web content to improve the relevance and accuracy of search results within Claude.
How can I block Claude-SearchBot?
You can block Claude-SearchBot by adding a Disallow rule for the user-agent Claude-SearchBot in your site's robots.txt file.
Does Claude-SearchBot follow robots.txt rules?
Yes, Claude-SearchBot honors the Robots Exclusion Protocol, so it will respect the directives you set in robots.txt.
Will blocking Claude-SearchBot affect my site's visibility in Claude?
Blocking Claude-SearchBot can reduce your site's visibility and accuracy in Claude search results, which may lead to less traffic from Claude users.
Is Claude-SearchBot used for training AI models?
No, Claude-SearchBot is used only for indexing content to improve search results in Claude, not for training language models.
Data & Sources
- Anthropic documentation - Primary source for Claude-SearchBot crawler details.
- Claude-SearchBot verification: Published IP ranges - Anthropic says IP blocking is not a durable opt-out because it can prevent the crawler from reading robots.txt.
- Claude-SearchBot live crawler data - Trakkr crawler telemetry for this user agent.