# What is Claude-SearchBot? AI crawler guide

Canonical URL: https://trakkr.ai/bots/claude-searchbot
Published: 2026-06-11
Last updated: 2026-06-11

Learn what Claude-SearchBot is, who operates it, its verified user-agent, robots.txt posture, and how blocking it can affect AI search, citations, training, or agent visibility.

Anthropic search crawler that indexes content to improve Claude search result relevance and accuracy.

## What is Claude-SearchBot?

Claude-SearchBot is a web crawler operated by Anthropic. It indexes content to improve the relevance and accuracy of search results within Claude. The bot identifies itself with the user-agent token Claude-SearchBot and respects the Robots Exclusion Protocol. It is distinct from other Anthropic crawlers and is used solely for search indexing purposes. Site owners can control its access through standard robots.txt directives. It does not collect data for training language models, only for enhancing search functionality within Claude.

## What it's for

If you run a website, allowing Claude-SearchBot can help your content appear in Claude search results, potentially increasing visibility to users who rely on Claude for information. Blocking it may reduce your site's presence and accuracy in those results, which could affect traffic from Claude users.

## How to handle Claude-SearchBot

To manage Claude-SearchBot, add a rule in your robots.txt file targeting the user-agent Claude-SearchBot. You can disallow specific paths or the entire site. The bot honors robots.txt directives, so this is the primary method to control its crawling.

## robots.txt rule

User-agent: Claude-SearchBot
Disallow: /

## Blocking cost

Blocking Claude-SearchBot can reduce your site's visibility and accuracy in Claude search results, potentially limiting traffic from users who rely on Claude for information.

## Examples

- A news website allows Claude-SearchBot to crawl its articles, leading to those articles appearing in Claude search results when users ask about current events.
- An e-commerce site blocks Claude-SearchBot via robots.txt, and its product pages no longer show up in Claude search results, reducing referral traffic.
- A blog partially allows Claude-SearchBot by disallowing certain directories, resulting in only selected posts being indexed and surfaced in Claude searches.

## Related bots

- OAI-SearchBot: Also tracked as a search crawler.
- Meta-WebIndexer: Also tracked as a search crawler.
- MistralAI-Index: Also tracked as a search crawler.
- PerplexityBot: Also tracked as a search crawler.
- Applebot: Also tracked as a search crawler.
- Bravebot: Also tracked as a search crawler.
- YouBot: Also tracked as a search crawler.
- PhindBot: Also tracked as a search crawler.
- YandexAdditional: Also tracked as a search crawler.
- Anthropic: Claude-SearchBot connects this operator term to its crawler behavior.
- Claude: Claude-SearchBot gives crawler context for Claude.

## Frequently Asked Questions

### What does Claude-SearchBot do?

Claude-SearchBot is a crawler from Anthropic that indexes web content to improve the relevance and accuracy of search results within Claude.

### How can I block Claude-SearchBot?

You can block Claude-SearchBot by adding a Disallow rule for the user-agent Claude-SearchBot in your site's robots.txt file.

### Does Claude-SearchBot follow robots.txt rules?

Yes, Claude-SearchBot honors the Robots Exclusion Protocol, so it will respect the directives you set in robots.txt.

### Will blocking Claude-SearchBot affect my site's visibility in Claude?

Blocking Claude-SearchBot can reduce your site's visibility and accuracy in Claude search results, which may lead to less traffic from Claude users.

### Is Claude-SearchBot used for training AI models?

No, Claude-SearchBot is used only for indexing content to improve search results in Claude, not for training language models.

## Data And Sources

- [Anthropic documentation](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) - Primary source for Claude-SearchBot crawler details.
- [Claude-SearchBot live crawler data](https://trakkr.ai/data/crawlers/claude-searchbot) - Trakkr crawler telemetry for this user agent.