What is DeepSeekBot? AI crawler guide

DeepSeekBot by DeepSeek: DeepSeek crawler token reported for training language models and improving AI products. Check its reported user-agent, robots.txt behavior, source, and verification guidance.

DeepSeek crawler token reported for training language models and improving AI products.

What is DeepSeekBot?

DeepSeekBot is a web crawler operated by DeepSeek. It is reported to collect publicly accessible web content for training language models and improving AI products. The crawler identifies itself with the user-agent token DeepSeekBot. No first-party documentation from DeepSeek was available at the time of this writing, so the information here comes from a public crawler registry. Because its purpose is tied to AI training, site owners may want to manage its access to their content.

What it's for

If you run a website, DeepSeekBot may visit your pages to gather data that could be used to train DeepSeek's language models or enhance its AI offerings. Allowing it could mean your content contributes to those systems. Disallowing it signals that you prefer your content not be used for that purpose, though the registry notes that compliance with robots.txt directives may not be reliable.

Allowing DeepSeekBot creates the possibility of retrieval. To find out whether DeepSeek actually cites or recommends you, track your DeepSeek mentions over time.

How to handle DeepSeekBot

To block DeepSeekBot, add a robots.txt rule targeting the user-agent token DeepSeekBot. Because compliance is reported as partial, this may reduce but not guarantee that the crawler will stop visiting your site. Monitor your server logs for the full user-agent string DeepSeekBot to confirm whether the rule is being honored.

robots.txt rule

User-agent: DeepSeekBot Disallow: /

Blocking cost

Blocking DeepSeekBot may prevent your content from being used in DeepSeek's AI training or product improvements, but it could also mean your site is not represented in any AI-generated answers, citations, or agent interactions that rely on that data.

Examples

• A blog author who does not want their writing used to train AI models might disallow DeepSeekBot in robots.txt.
• An e-commerce site that relies on AI-driven product recommendations could allow DeepSeekBot to help improve those systems.
• A news publisher may block DeepSeekBot to keep their articles out of AI training datasets, though the crawler might still access the content if compliance is inconsistent.

Related bots

• Bytespider: Also tracked as a training crawler.
• cohere-training-data-crawler: Also tracked as a training crawler.
• LAIONDownloader: Also tracked as a training crawler.
• PanguBot: Also tracked as a training crawler.
• Meta-ExternalAgent: Also tracked as a training crawler.
• Google-Extended: Also tracked as a training crawler.
• GPTBot: Also tracked as a training crawler.
• img2dataset: Also tracked as a training crawler.
• Ai2Bot-Dolma: Also tracked as a training crawler.
• AI Training Opt-Out: DeepSeekBot is a training crawler tied to this policy decision.
• Robots.txt: Robots.txt is the control file used to allow or block DeepSeekBot.