What is Ai2Bot-Dolma? AI crawler guide

Ai2Bot-Dolma by Allen Institute for AI: AI2 crawler token associated with Dolma/open language model dataset collection. Check its reported user-agent, robots.txt behavior, source, and verification guidance.

AI2 crawler token associated with Dolma/open language model dataset collection.

What is Ai2Bot-Dolma?

Ai2Bot-Dolma is a web crawler operated by the Allen Institute for AI. It is associated with the Dolma project, which collects open language model training datasets. The bot identifies itself with the user-agent token Ai2Bot-Dolma and honors robots.txt directives. Its purpose is to gather publicly available web content for use in AI research and dataset construction.

What it's for

If you allow Ai2Bot-Dolma, your site's content may be included in the Dolma dataset, which could be used to train open language models. Blocking it prevents your content from being collected for this specific AI2 training data initiative.

Ai2Bot-Dolma collects pages for model training. Training inclusion is not the same as being cited, so measure where AI answers actually cite your site before drawing conclusions from crawl logs.

How to handle Ai2Bot-Dolma

To prevent Ai2Bot-Dolma from crawling your site, add a disallow rule for the user-agent token Ai2Bot-Dolma in your robots.txt file. The bot respects standard robots.txt directives.

robots.txt rule

User-agent: Ai2Bot-Dolma Disallow: /

Blocking cost

Blocking Ai2Bot-Dolma may exclude your site's content from the Dolma dataset, potentially reducing its representation in AI2's open language model training data.

Examples

• Ai2Bot-Dolma crawls a news website and collects article text for the Dolma dataset.
• The bot visits a public research paper repository and downloads PDFs for language model training.
• Ai2Bot-Dolma accesses a blog and extracts post content to include in open dataset collection.

Related bots

• AI2Bot: Another Allen Institute for AI training crawler to compare.
• ICC-Crawler: Also tracked as a training crawler.
• img2dataset: Also tracked as a training crawler.
• LAIONDownloader: Also tracked as a training crawler.
• PanguBot: Also tracked as a training crawler.
• CCBot: Also tracked as a training crawler.
• Google-Extended: Also tracked as a training crawler.
• Webzio-Extended: Also tracked as a training crawler.
• cohere-training-data-crawler: Also tracked as a training crawler.
• AI Training Opt-Out: Ai2Bot-Dolma is a training crawler tied to this policy decision.
• Robots.txt: Robots.txt is the control file used to allow or block Ai2Bot-Dolma.