What is Claude-Code? AI crawler guide
Claude-Code by Anthropic: Claude Code related agent token seen in crawler/user-agent lists. Check its reported user-agent, robots.txt behavior, source, and verification guidance.
Claude Code related agent token seen in crawler/user-agent lists.
What is Claude-Code?
Claude-Code is a user-agent token associated with Anthropic's Claude Code developer tool. It appears in crawler and user-agent lists as an interaction bot, meaning it is typically used for developer-tool retrieval rather than broad web crawling. The token is unverified in terms of its robots.txt compliance posture, and no official documentation from Anthropic confirms its behavior. Site owners may encounter requests from this agent when developers use Claude Code to fetch or interact with web resources programmatically.
What it's for
For a site owner, a hit from Claude-Code likely indicates a developer using Anthropic's tool to retrieve content for coding assistance or integration, not a traditional search crawler. This means the impact on server resources is probably minimal and targeted, but you should verify the request source before creating access policies to ensure it aligns with your site's intended use.
Claude-Code fetches pages for a developer tool, so its visits rarely map to buyer behavior. For the demand side, see how much human traffic AI platforms actually send you.
How to handle Claude-Code
If you wish to block Claude-Code, add a robots.txt entry disallowing the user-agent token. Since its compliance posture is unverified, blocking may or may not be honored. Monitor logs for this token to understand request patterns, and consider reaching out to Anthropic for clarification if needed.
robots.txt rule
User-agent: Claude-Code Disallow: /
Blocking cost
Blocking Claude-Code could prevent your content from being used in developer tool interactions, potentially reducing visibility in AI-assisted coding workflows.
Examples
- A developer uses Claude Code to fetch documentation from a public API, and the request appears with the Claude-Code user-agent.
- Claude-Code retrieves a snippet of open-source code from a repository to assist in a coding task.
- The token shows up in server logs when a user asks Claude Code to summarize a publicly accessible web page.
Related bots
- Google-Gemini-CLI: Also tracked as a live fetcher.
- Claude-Web: Another Anthropic live fetcher to compare.
- Google-Agent: Also tracked as a live fetcher.
- Meta-ExternalFetcher: Also tracked as a live fetcher.
- cohere-ai: Also tracked as a live fetcher.
- Google-GeminiNotebook: Also tracked as a live fetcher.
- GoogleAgent-Mariner: Also tracked as a live fetcher.
- NovaAct: Also tracked as a live fetcher.
- ChatGPT Agent: Also tracked as a live fetcher.
- Anthropic: Claude-Code connects this operator term to its crawler behavior.
- Claude 3.5 Sonnet: Claude-Code gives crawler context for Claude 3.5 Sonnet.
Frequently Asked Questions
Is Claude-Code an official Anthropic crawler?
Claude-Code is an unverified token associated with Anthropic's Claude Code tool, but no official documentation confirms its behavior or compliance.
Does Claude-Code obey robots.txt?
The robots.txt compliance posture of Claude-Code is unverified, so it is unclear whether it will honor disallow directives.
What should I do if I see Claude-Code in my logs?
Validate the request source and pattern. Since it is likely developer-tool retrieval, assess whether the traffic aligns with your site's policies before blocking.
Can blocking Claude-Code affect my site's visibility in AI tools?
Blocking may prevent your content from being used in Claude Code interactions, which could reduce visibility in AI-assisted developer workflows.
Data & Sources
- Claude-Code source reference - Source used to verify Claude-Code.