What is Claude-Web? AI crawler guide

Claude-Web by Anthropic: Reported Anthropic-related token seen in public crawler registries and Trakkr detection, but absent from Anthropic's current bot documentation. Check its reported user-agent, robots.txt behavior, source, and verification guidance.

Reported Anthropic-related token seen in public crawler registries and Trakkr detection, but absent from Anthropic's current bot documentation.

What is Claude-Web?

Claude-Web is a reported user-agent token associated with Anthropic in public crawler registries and Trakkr detection. Anthropic does not list it in its current crawler guidance. That guidance names ClaudeBot for potential model training, Claude-SearchBot for search indexing, and Claude-User for user-directed retrieval. Claude-Web's operator, purpose, and robots.txt behavior therefore remain unverified.

What it's for

A Claude-Web string in a log is evidence of a matching user-agent, not proof that Anthropic sent the request. It should not be counted as a training, search, retrieval, or conversation signal without another identity check. Use Anthropic's three documented tokens when making visibility or training-policy decisions.

Claude-Web fires when a real user's question pulls in your page. To know how often Claude reaches for your brand at all, track your brand mentions in Claude.

How to handle Claude-Web

First, keep the raw request details and check whether the source IP appears in Anthropic's published crawler list. A Claude-Web robots.txt rule can express your preference, but there is no operator documentation confirming that this token reads or follows the rule. Apply normal edge and application controls when access must be enforced.

robots.txt rule

User-agent: Claude-Web Disallow: /

Blocking cost

The effect of blocking Claude-Web is unknown because Anthropic does not document the token. Do not infer an effect on Claude training, search, or user-directed retrieval from this rule alone.

Examples

A site owner notices Claude-Web in server logs but finds no mention of it on Anthropic's official crawler documentation.
A webmaster adds a blanket block for Claude-Web in robots.txt while still allowing ClaudeBot for AI training opt-in.
A developer checks public crawler registries and sees Claude-Web listed alongside other Anthropic tokens, but with no usage details.

Related bots

Google-Gemini-CLI: Also tracked as a live fetcher.
Claude-Code: Another Anthropic live fetcher to compare.
AmazonBuyForMe: Also tracked as a live fetcher.
GoogleAgent-Mariner: Also tracked as a live fetcher.
ChatGPT Agent: Also tracked as a live fetcher.
Google-Agent: Also tracked as a live fetcher.
cohere-ai: Also tracked as a live fetcher.
Claude-User: Another Anthropic live fetcher to compare.
Gemini-Deep-Research: Also tracked as a live fetcher.
Anthropic-AI: Claude-Web connects this operator term to its crawler behavior.
AI Crawlers: Claude-Web is a concrete crawler example for this concept.

Frequently Asked Questions

Is Claude-Web an official Anthropic crawler?

Anthropic has not published documentation for Claude-Web. Its current public guidance covers ClaudeBot, Claude-SearchBot, and Claude-User, so Claude-Web's status is unverified.

Does Claude-Web follow robots.txt?

There is no official statement from Anthropic about Claude-Web's robots.txt compliance. Because the token is unverified, its behavior cannot be guaranteed.

Will blocking Claude-Web affect my site in Claude's AI products?

The effect is unknown. Anthropic documents separate ClaudeBot, Claude-SearchBot, and Claude-User controls, so do not infer that a Claude-Web rule changes those systems.

How is Claude-Web different from ClaudeBot?

ClaudeBot is a documented crawler used for AI training and product improvement, with clear policy guidance. Claude-Web is an unverified token with no official description, so its purpose and behavior are unknown.

Should I block Claude-Web?

Choose based on your access policy and enforce it at the edge if needed. A robots.txt rule can express a preference, but there is no operator documentation confirming Claude-Web behavior.

Data & Sources

Claude-Web source reference - Source used to verify Claude-Web.