What is Claude-User? AI crawler guide
Claude-User by Anthropic: Anthropic user-triggered fetcher for Claude answers that need a specific web page. Check its reported user-agent, robots.txt behavior, source, and verification guidance.
Anthropic user-triggered fetcher for Claude answers that need a specific web page.
What is Claude-User?
Claude-User is a web fetcher operated by Anthropic. It acts on behalf of a user who asks Claude a question that requires fetching a specific web page. Unlike broad crawlers that scan the web for training data, Claude-User retrieves content only when a user directly requests it. The bot identifies itself with the user-agent token Claude-User and honors robots.txt directives. This means site owners can control whether Claude-User accesses their pages by setting rules in their robots.txt file.
What it's for
If you allow Claude-User, your content can appear in Claude's answers when a user asks a question that needs your page. This may drive traffic and visibility from users who rely on Claude for information. Blocking it prevents Claude from retrieving your content in response to user-directed queries, which could reduce your site's exposure to Claude's user base.
Claude-User fires when a real user's question pulls in your page. To know how often Claude reaches for your brand at all, track your brand mentions in Claude.
How to handle Claude-User
To block Claude-User, add a robots.txt rule targeting the user-agent token Claude-User. The bot respects standard robots.txt directives, so a simple disallow rule will prevent it from fetching your pages. If you want to allow access, ensure no such rule exists or explicitly allow it.
robots.txt rule
User-agent: Claude-User Disallow: /
Blocking cost
Blocking Claude-User may prevent your content from being retrieved and shown in Claude's answers when users ask about your site, potentially reducing visibility and traffic from Claude's users.
Examples
- A user asks Claude for the latest product details from a specific online store. Claude-User fetches the product page and includes the information in its response.
- Someone queries Claude about a recent news article. Claude-User retrieves the article from the news site to provide a summary.
- A developer asks Claude to check the documentation for a software library. Claude-User fetches the relevant documentation page to answer the question.
Related bots
- DuckAssistBot: Also tracked as a live fetcher.
- Gemini-Deep-Research: Also tracked as a live fetcher.
- MistralAI-User: Also tracked as a live fetcher.
- cohere-ai: Also tracked as a live fetcher.
- Amzn-User: Also tracked as a live fetcher.
- ChatGPT-User: Also tracked as a live fetcher.
- Kimi-User: Also tracked as a live fetcher.
- Perplexity-User: Also tracked as a live fetcher.
- Claude-Code: Another Anthropic live fetcher to compare.
- Anthropic: Claude-User connects this operator term to its crawler behavior.
- Claude 3.5 Sonnet: Claude-User gives crawler context for Claude 3.5 Sonnet.
- SemrushBot-OCOB: Shares crawler-policy context with Claude-User.
Frequently Asked Questions
Does Claude-User crawl the web for training data?
No. Claude-User only fetches specific web pages when a user asks Claude a question that requires that page. It is not used for broad crawling or training data collection.
How can I block Claude-User from accessing my site?
You can block Claude-User by adding a disallow rule for the user-agent token Claude-User in your robots.txt file. The bot honors standard robots.txt directives.
Will blocking Claude-User affect my site's appearance in other search engines?
Blocking Claude-User only affects whether Claude can retrieve your content in response to user queries. It does not impact your site's indexing or ranking in other search engines.
Does Claude-User respect robots.txt?
Yes, Claude-User honors robots.txt directives. If you disallow it, the bot will not fetch your pages.
Data & Sources
- Anthropic documentation - Primary source for Claude-User crawler details.
- Claude-User verification: Published IP ranges - Anthropic says IP blocking is not a durable opt-out because it can prevent the crawler from reading robots.txt.