What is PanguBot? AI crawler guide

PanguBot by Huawei: Huawei crawler token reported for training data collection for the PanGu multimodal LLM. Check its reported user-agent, robots.txt behavior, source, and verification guidance.

Huawei crawler token reported for training data collection for the PanGu multimodal LLM.

What is PanguBot?

PanguBot is a web crawler operated by Huawei. It is reported to collect training data for the PanGu multimodal large language model. The crawler identifies itself with the user-agent token PanguBot. Its presence in a community-maintained repository suggests it is used for large-scale data gathering to support Huawei's AI development efforts. The bot's behavior and compliance with robots.txt are not independently verified, so its exact crawling patterns remain unclear. Site owners may encounter it as part of Huawei's data collection infrastructure for training their AI models.

What it's for

For a site owner, PanguBot represents Huawei's data collection for training its PanGu AI model. Allowing it may contribute your content to Huawei's model training, potentially influencing AI-generated outputs. Blocking it serves as an opt-out from this specific collection, which could help control how your content is used in AI training by Huawei.

PanguBot collects pages for model training. Training inclusion is not the same as being cited, so measure where AI answers actually cite your site before drawing conclusions from crawl logs.

How to handle PanguBot

To block PanguBot, add a robots.txt rule targeting the user-agent token PanguBot with a Disallow directive. Since its compliance is unverified, blocking may not be guaranteed, but it is the standard method to express your opt-out preference. Monitor your server logs for actual PanguBot activity to confirm the block's effectiveness.

robots.txt rule

User-agent: PanguBot Disallow: /

Blocking cost

Blocking PanguBot may prevent your content from being used in Huawei's PanGu model training, which could reduce your visibility in AI-generated answers or citations from that model.

Examples

• A news website notices PanguBot in its logs and blocks it to prevent articles from being used in PanGu training.
• An e-commerce site allows PanguBot, hoping product descriptions might appear in AI-generated shopping recommendations.
• A blog adds a robots.txt rule for PanguBot after learning it may collect data for Huawei's AI, opting out of training use.

Related bots

• cohere-training-data-crawler: Also tracked as a training crawler.
• DeepSeekBot: Also tracked as a training crawler.
• Ai2Bot-Dolma: Also tracked as a training crawler.
• Applebot-Extended: Also tracked as a training crawler.
• GPTBot: Also tracked as a training crawler.
• KimiBot: Also tracked as a training crawler.
• LAIONDownloader: Also tracked as a training crawler.
• Meta-ExternalAgent: Also tracked as a training crawler.
• TerraCotta: Also tracked as a training crawler.
• AI Training Opt-Out: PanguBot is a training crawler tied to this policy decision.
• Robots.txt: Robots.txt is the control file used to allow or block PanguBot.