What is Meta-ExternalFetcher? AI crawler guide

Meta-ExternalFetcher by Meta: Meta user-requested fetcher for AI and link features across Meta products. Check its reported user-agent, robots.txt behavior, source, and verification guidance.

Meta user-requested fetcher for AI and link features across Meta products.

What is Meta-ExternalFetcher?

Meta-ExternalFetcher is a web crawler operated by Meta. It fetches specific URLs on behalf of Meta users, rather than performing broad, autonomous crawling. When a user takes an action within a Meta product-such as sharing a link, asking an AI assistant to retrieve information, or using a feature that needs to preview a webpage-this bot may be dispatched to fetch that exact URL. It is documented by Meta as a user-requested fetcher, meaning its visits are typically tied to direct user intent. The bot identifies itself with the user-agent token Meta-ExternalFetcher and is part of Meta's suite of crawlers that support link sharing, AI features, and other interactive services across platforms like Facebook, Instagram, and WhatsApp.

What it's for

For site owners, a visit from Meta-ExternalFetcher usually means a Meta user has triggered a fetch of a specific page on your site. This could happen when someone shares your link in a Meta product, asks a Meta AI assistant about your content, or uses a feature that generates a link preview. Unlike broad crawlers that scan many pages, this bot is on-demand and targeted. Its activity can drive referral traffic and increase visibility within Meta's ecosystem, but it may also consume server resources for individual requests. Understanding this bot helps you distinguish user-driven fetches from automated indexing crawls.

Meta-ExternalFetcher fires when a real user's question pulls in your page. To know how often Meta AI reaches for your brand at all, track your brand mentions in Meta AI.

How to handle Meta-ExternalFetcher

You can block Meta-ExternalFetcher using robots.txt, but Meta notes that because it performs user-requested fetches, it may bypass robots.txt restrictions. If you want to prevent Meta from fetching your content in response to user actions, a robots.txt disallow rule may not be fully effective. If you are comfortable with user-driven fetches, no action is needed; the bot will fetch URLs only when a user action triggers it.

robots.txt rule

User-agent: Meta-ExternalFetcher Disallow: /

Blocking cost

Blocking Meta-ExternalFetcher may prevent your content from appearing in user-requested link previews, AI responses, or other interactive features within Meta products, potentially reducing referral traffic and visibility.

Examples

Related bots

Frequently Asked Questions

Does Meta-ExternalFetcher obey robots.txt?

Meta documents that Meta-ExternalFetcher may bypass robots.txt because it performs user-requested fetches. A disallow rule might not prevent it from accessing your site.

How is Meta-ExternalFetcher different from other Meta crawlers?

Unlike broad crawlers that scan many pages for indexing, Meta-ExternalFetcher fetches specific URLs only when a user action triggers it, such as sharing a link or using an AI feature.

Will blocking Meta-ExternalFetcher affect my site's search ranking?

Blocking this bot does not directly impact search engine rankings, but it may prevent your content from appearing in user-requested link previews or AI responses within Meta products.

Can I verify that a visit is really from Meta-ExternalFetcher?

You can check the user-agent string and perform reverse DNS lookups to confirm the request originates from Meta's IP ranges, as documented in their official web crawler information.

Does Meta-ExternalFetcher cache fetched content?

Meta may temporarily cache fetched content to improve performance for link previews and other features, but the specifics of caching are not detailed in their public documentation.

Data & Sources