What is anthropic-ai? AI crawler guide
anthropic-ai by Anthropic: Legacy Anthropic robots.txt token that predates the current ClaudeBot, Claude-User and Claude-SearchBot names. Check its reported user-agent, robots.txt behavior, source, and verification guidance.
Legacy Anthropic robots.txt token that predates the current ClaudeBot, Claude-User and Claude-SearchBot names.
What is anthropic-ai?
The anthropic-ai token is a legacy robots.txt user-agent identifier that predates Anthropic's current crawler names. It was once associated with Anthropic's web crawling activities but is no longer listed in the company's official documentation. The token remains widely present in existing robots.txt files, yet its current use is unverified. Any traffic observed with this user-agent should be confirmed before it is treated as originating from Anthropic, as it may not correspond to any actively documented crawler.
What it's for
For site owners, the presence of this token in robots.txt is unlikely to control any known Anthropic crawler, since Anthropic's current guidance does not reference it. Relying on it for access management could leave content exposed to Anthropic's documented crawlers, which use different identifiers. Site owners should review their robots.txt rules to ensure they target the correct, currently documented user-agent tokens if they wish to manage Anthropic's access.
How to handle anthropic-ai
Verify any traffic using the anthropic-ai user-agent before taking action, as it may not be from Anthropic. To block Anthropic's known crawlers, use the tokens ClaudeBot, Claude-User, and Claude-SearchBot as specified in Anthropic's official documentation.
robots.txt rule
User-agent: anthropic-ai Disallow: /
Blocking cost
Blocking the anthropic-ai token is unlikely to affect visibility in AI-powered search, answers, citations, or training data, because it is not associated with any currently documented Anthropic crawler.
Examples
- A site owner adds a Disallow rule for anthropic-ai, but later discovers that ClaudeBot continues to crawl the site because the rule does not match the active crawler's user-agent.
- Logs show requests from a user-agent string containing only 'anthropic-ai', but the IP ranges do not match known Anthropic infrastructure, suggesting the traffic may be from a third party or misconfigured tool.
- During a robots.txt audit, a webmaster finds an old anthropic-ai rule and replaces it with rules for ClaudeBot, Claude-User, and Claude-SearchBot to align with current Anthropic documentation.
Related bots
- Applebot-Extended: Also tracked as a training crawler.
- Google-Extended: Also tracked as a training crawler.
- GPTBot: Also tracked as a training crawler.
- Meta-ExternalAgent: Also tracked as a training crawler.
- ClaudeBot: Another Anthropic training crawler to compare.
- DeepSeekBot: Also tracked as a training crawler.
- cohere-training-data-crawler: Also tracked as a training crawler.
- img2dataset: Also tracked as a training crawler.
- PanguBot: Also tracked as a training crawler.
- Anthropic-AI: Anthropic-AI is the glossary definition behind this crawler guide.
- AI Training Opt-Out: anthropic-ai is a training crawler tied to this policy decision.
Frequently Asked Questions
Is anthropic-ai still used by Anthropic?
Anthropic's current documentation does not list anthropic-ai as an active crawler token. Any traffic using this identifier should be verified, as it may not originate from Anthropic.
Will blocking anthropic-ai stop Anthropic from crawling my site?
No, because Anthropic's documented crawlers use different tokens such as ClaudeBot, Claude-User, and Claude-SearchBot. A rule for anthropic-ai will not affect those crawlers.
Should I remove the anthropic-ai rule from my robots.txt?
It is safe to remove or keep the rule, but it should not be your only method for controlling Anthropic's access. Ensure you have rules for the currently documented tokens if you wish to manage crawling.
How can I identify if traffic from anthropic-ai is legitimate?
Check the IP addresses against known Anthropic ranges and review the request patterns. If they do not match expected behavior, the traffic may be from a different source.
Data & Sources
- Anthropic documentation - Primary source for anthropic-ai crawler details.
- anthropic-ai source reference - Source used to verify anthropic-ai.
- anthropic-ai verification: Anthropic published IP ranges - Trakkr measured 20,197 anthropic-ai requests carrying a source IP across its monitored sites in the 30 days to 11 August 2026. None fell inside Anthropic's published ranges, while the same test matched 48% of ClaudeBot and 24% of Claude-User requests.