What is MistralAI-Training? AI crawler guide
MistralAI-Training by Mistral AI: Mistral crawler for content that may be used to train its models, kept separate from its search index crawler. Check its reported user-agent, robots.txt behavior, source, and verification guidance.
Mistral crawler for content that may be used to train its models, kept separate from its search index crawler.
What is MistralAI-Training?
MistralAI-Training is a web crawler operated by Mistral AI. It collects publicly accessible web content that may be used to train Mistral's AI models. The crawler is kept separate from MistralAI-Index, the company's search index crawler, so that training and retrieval functions can be controlled independently. It identifies itself with the user-agent token MistralAI-Training and honors robots.txt directives. The crawler's full user-agent string includes a reference to Mistral's documentation on crawler behavior.
What it's for
For site owners, MistralAI-Training represents a way for Mistral AI to gather training data from the open web. Allowing the crawler means your content could help improve Mistral's models. Blocking it lets you opt out of having your content used for training while still remaining visible in Mistral's search and answer products through the separate MistralAI-Index crawler.
How to handle MistralAI-Training
To prevent MistralAI-Training from accessing your site, add a robots.txt rule that disallows the user-agent token MistralAI-Training. This will stop the crawler from fetching your pages for training purposes. The separate MistralAI-Index crawler can still be allowed if you want your content to appear in Le Chat search results.
robots.txt rule
User-agent: MistralAI-Training Disallow: /
Blocking cost
Blocking MistralAI-Training may prevent your content from being used to train Mistral's models, but it does not affect your visibility in Mistral's search or answer features, which rely on the separate MistralAI-Index crawler.
Examples
- A news website allows MistralAI-Training, and its articles may be used to improve Mistral's language models.
- A blog disallows MistralAI-Training in robots.txt, so its posts are not used for training, but the blog can still appear in Le Chat search results via MistralAI-Index.
- An e-commerce site blocks MistralAI-Training to keep its product descriptions out of training data while remaining discoverable through Mistral's search products.
Related bots
- GPTBot: Also tracked as a training crawler.
- KimiBot: Also tracked as a training crawler.
- Meta-ExternalAgent: Also tracked as a training crawler.
- Bytespider: Also tracked as a training crawler.
- Applebot-Extended: Also tracked as a training crawler.
- CCBot: Also tracked as a training crawler.
- AI2Bot: Also tracked as a training crawler.
- LAIONDownloader: Also tracked as a training crawler.
- ClaudeBot: Also tracked as a training crawler.
- AI Training Opt-Out: MistralAI-Training is a training crawler tied to this policy decision.
- Robots.txt: Robots.txt is the control file used to allow or block MistralAI-Training.
Frequently Asked Questions
What is the difference between MistralAI-Training and MistralAI-Index?
MistralAI-Training is used to collect content that may be used to train Mistral's AI models. MistralAI-Index is a separate crawler for building a search index so that content can appear in Le Chat search results. They are controlled independently via robots.txt.
Does blocking MistralAI-Training affect my site's visibility in Le Chat?
No. Blocking MistralAI-Training only opts your content out of training use. Your site can still be indexed and appear in Le Chat search results if you allow the MistralAI-Index crawler.
How do I block MistralAI-Training?
Add a robots.txt rule that disallows the user-agent token MistralAI-Training. This tells the crawler not to access any part of your site for training purposes.
Does MistralAI-Training honor robots.txt?
Yes, MistralAI-Training honors robots.txt directives. If you disallow it, the crawler will not fetch your content.
Data & Sources
- Mistral AI documentation - Primary source for MistralAI-Training crawler details.