What is ICC-Crawler? AI crawler guide

ICC-Crawler by NICT: NICT crawler for data used in artificial intelligence technologies and third-party research/commercial uses. Check its reported user-agent, robots.txt behavior, source, and verification guidance.

NICT crawler for data used in artificial intelligence technologies and third-party research/commercial uses.

What is ICC-Crawler?

ICC-Crawler is a web crawler operated by NICT, Japan's National Institute of Information and Communications Technology. It collects publicly available web data to build datasets used in artificial intelligence technologies and for third-party research and commercial applications. The crawler identifies itself with the user-agent token ICC-Crawler and respects the Robots Exclusion Protocol, meaning it will obey directives placed in a site's robots.txt file. Its activity is focused on gathering text and other content that can support machine learning model development and broader AI research initiatives.

What it's for

For a site owner, ICC-Crawler's visits mean your content may be included in datasets that fuel AI development and research, potentially benefiting innovation but also raising considerations about how your data is used downstream. Allowing the crawler can contribute to scientific and commercial AI advancements, while blocking it ensures your site's content is not harvested for these purposes.

ICC-Crawler collects pages for model training. Training inclusion is not the same as being cited, so measure where AI answers actually cite your site before drawing conclusions from crawl logs.

How to handle ICC-Crawler

To prevent ICC-Crawler from accessing your site, add a disallow rule for the user-agent token ICC-Crawler in your robots.txt file. Because the crawler honors robots.txt, this will stop it from crawling any disallowed paths. If you wish to permit crawling, simply omit the rule or explicitly allow it.

robots.txt rule

User-agent: ICC-Crawler Disallow: /

Blocking cost

Blocking ICC-Crawler may prevent your site's content from appearing in AI training datasets and research corpora, which could reduce its potential influence on future AI systems and limit visibility in AI-driven applications.

Examples

• ICC-Crawler visits a news website and collects article text for inclusion in a multilingual language model training set.
• A research institution allows ICC-Crawler to gather open-access papers, which are then used to improve academic search and summarization tools.
• An e-commerce site blocks ICC-Crawler, so its product descriptions are not included in a dataset for training product recommendation algorithms.

Related bots

• CCBot: Also tracked as a training crawler.
• Ai2Bot-Dolma: Also tracked as a training crawler.
• LAIONDownloader: Also tracked as a training crawler.
• AI2Bot: Also tracked as a training crawler.
• Google-Extended: Also tracked as a training crawler.
• Webzio-Extended: Also tracked as a training crawler.
• img2dataset: Also tracked as a training crawler.
• ClaudeBot: Also tracked as a training crawler.
• GPTBot: Also tracked as a training crawler.
• Robots.txt: Robots.txt is the control file used to allow or block ICC-Crawler.
• AI Training Opt-Out: ICC-Crawler is a training crawler tied to this policy decision.