What is Meta-ExternalAgent? AI crawler guide
Meta-ExternalAgent by Meta: Meta crawler for indexing content directly for AI model training and product improvement use cases. Check its reported user-agent, robots.txt behavior, source, and verification guidance.
Meta crawler for indexing content directly for AI model training and product improvement use cases.
What is Meta-ExternalAgent?
Meta-ExternalAgent is a web crawler operated by Meta that collects publicly accessible content from the web. Its primary purpose is to index information directly for AI model training and product improvement use cases. The crawler identifies itself with the user-agent token Meta-ExternalAgent and honors robots.txt directives. It is distinct from other Meta fetchers that may be triggered by user actions, focusing instead on autonomous crawling for AI development. Site owners can control its access through standard robots.txt rules, and Meta provides official documentation detailing its behavior and how to manage it.
What it's for
For site owners, Meta-ExternalAgent represents a direct path through which Meta may use your site's content to train AI models and improve its products. Allowing the crawler could mean your public content contributes to Meta's AI systems, while blocking it opts your site out of this specific crawling channel. However, blocking does not prevent all Meta-related access, as other user-triggered fetchers may still retrieve content when users interact with Meta services.
Allowing Meta-ExternalAgent creates the possibility of retrieval. To find out whether Meta AI actually cites or recommends you, track your Meta AI mentions over time.
How to handle Meta-ExternalAgent
To prevent Meta-ExternalAgent from crawling your site, add a robots.txt rule that disallows the user-agent token Meta-ExternalAgent. This instructs the crawler to skip your site entirely. Because the crawler honors robots.txt, the block is effective for this specific agent. Keep in mind that this only affects the direct AI training crawl path and does not stop other Meta bots that may fetch content on behalf of user actions.
robots.txt rule
User-agent: Meta-ExternalAgent Disallow: /
Blocking cost
Blocking Meta-ExternalAgent may prevent your site's content from being used in Meta's AI model training and product improvements, potentially reducing your visibility in AI-driven features or answers that rely on this crawl data.
Examples
- A news website allows Meta-ExternalAgent, and its articles may be used to help train Meta's language models, potentially influencing how those models generate text about current events.
- An e-commerce site blocks Meta-ExternalAgent via robots.txt, so its product descriptions are not directly crawled for AI training, though user-shared links on Meta platforms could still be fetched by other means.
- A blog that disallows Meta-ExternalAgent ensures its posts are not included in the autonomous crawl for AI improvement, but if a user shares a post on Facebook, a separate user-triggered fetcher might still retrieve it.
Related bots
- GPTBot: Also tracked as a training crawler.
- MistralAI-Training: Also tracked as a training crawler.
- Applebot-Extended: Also tracked as a training crawler.
- KimiBot: Also tracked as a training crawler.
- Google-Extended: Also tracked as a training crawler.
- CCBot: Also tracked as a training crawler.
- cohere-training-data-crawler: Also tracked as a training crawler.
- Webzio-Extended: Also tracked as a training crawler.
- LAIONDownloader: Also tracked as a training crawler.
- AI Training Opt-Out: Meta-ExternalAgent is a training crawler tied to this policy decision.
- Robots.txt: Robots.txt is the control file used to allow or block Meta-ExternalAgent.
- FacebookBot: Also operated by Meta.
Frequently Asked Questions
Does blocking Meta-ExternalAgent stop all Meta crawlers?
No, blocking Meta-ExternalAgent only affects this specific crawler used for direct AI training and product improvement. Other Meta bots, such as those triggered by user actions like sharing a link, may still access your content.
How can I verify that Meta-ExternalAgent is respecting my robots.txt?
You can check your server logs for requests from the user-agent token Meta-ExternalAgent. If you have disallowed it, you should see no successful fetches from this agent. Meta's documentation states that it honors robots.txt, so a proper disallow rule should prevent crawling.
Will blocking Meta-ExternalAgent affect my site's appearance in Meta products?
Blocking Meta-ExternalAgent does not directly impact how your site appears in Meta products like Facebook or Instagram. It only opts you out of the autonomous crawl for AI training. User-triggered fetches for link previews or shared content are handled by different crawlers.
Is Meta-ExternalAgent the only Meta crawler I need to manage for AI training?
Meta-ExternalAgent is the primary crawler for direct AI training and product improvement, but Meta may use other crawlers or data sources. Review Meta's official documentation for a complete list of crawlers and their purposes to ensure comprehensive control.
Data & Sources
- Meta documentation - Primary source for Meta-ExternalAgent crawler details.