What is ClaudeBot? AI crawler guide
ClaudeBot is a crawler or fetcher associated with Anthropic. See source-graded user-agent, robots.txt, verification, and observation guidance.
What is ClaudeBot?
ClaudeBot is a documented Anthropic crawler or fetcher. Anthropic crawler for public web content that could contribute to Claude model training.
Evidence status
| Field | Value |
|---|---|
| Evidence | Officially documented |
| Lifecycle | Active |
| Purpose | training |
| robots.txt posture | honors |
| Source checked | 2026-08-18 |
Documented user-agent
Mozilla/5.0 (compatible; ClaudeBot/1.0; [email protected])
Allowing ClaudeBot creates the possibility of retrieval. To find out whether Claude actually cites or recommends you, track your Claude mentions over time.
To see why ClaudeBot access is worth checking rather than assuming, read the fortnight one site spent unblocking crawlers, and what changed after.
robots.txt allow example
User-agent: ClaudeBot Allow: /
robots.txt block example
User-agent: ClaudeBot Disallow: /
What the rule can and cannot do
Robots.txt expresses an access policy to compliant automated crawlers. It does not authenticate the sender, remove content already collected, or guarantee that a model will use or not use content.
How to verify a request
- Match the complete documented user-agent where one exists. A match is only a clue.
- Use the operator-published check: Published IP ranges.
- Keep the observed request, operator documentation, and any inference as separate fields.
Observed in Trakkr connected-site data
| Measure | Value |
|---|---|
| Matching signature | ClaudeBot |
| Classified requests | 568865 |
| Sites in sample | 85 |
| Window | 2026-07-18 through 2026-08-17 |
| Method | Finalized daily crawler summaries from connected Trakkr sites, grouped by the crawler signature detected in each request. |
| Limits | This is a connected-site sample, not a representative sample of the web. A matching user-agent or signature does not prove that the named operator sent the request. Counts describe classified request signatures, not market share or unique pages crawled. |
JavaScript behavior
The cited operator material does not verify JavaScript rendering. Serve useful HTML before client-side JavaScript where possible.
Related bots
- Webzio-Extended: Also tracked as a training crawler.
- CCBot: Also tracked as a training crawler.
- GPTBot: Also tracked as a training crawler.
- KimiBot: Also tracked as a training crawler.
- AI2Bot: Also tracked as a training crawler.
- MistralAI-Training: Also tracked as a training crawler.
- VelenPublicWebCrawler: Also tracked as a training crawler.
- img2dataset: Also tracked as a training crawler.
- Bytespider: Also tracked as a training crawler.
- AI Training Opt-Out: ClaudeBot is a training crawler tied to this policy decision.
- Anthropic-AI: ClaudeBot connects this operator term to its crawler behavior.
Frequently Asked Questions
What is ClaudeBot?
ClaudeBot is a officially documented crawler or fetcher record associated with Anthropic.
What user-agent does ClaudeBot use?
Mozilla/5.0 (compatible; ClaudeBot/1.0; [email protected]). Check the evidence status before attributing a matching request.
Can ClaudeBot be blocked in robots.txt?
The record identifies ClaudeBot as the token to review. Robots.txt is a request policy, not proof of model use or non-use.
How can I verify a ClaudeBot request?
Start with the full user-agent, then use published ip ranges from the operator. A user-agent match alone is not proof.
Data & Sources
- Anthropic documentation - Primary source for ClaudeBot crawler details.
- ClaudeBot verification: Published IP ranges - Anthropic says IP blocking is not a durable opt-out because it can prevent the crawler from reading robots.txt.