What is Meta-WebIndexer? AI crawler guide

Meta-WebIndexer by Meta: Meta crawler for improving Meta AI search result quality and source linking. Check its reported user-agent, robots.txt behavior, source, and verification guidance.

Meta crawler for improving Meta AI search result quality and source linking.

What is Meta-WebIndexer?

Meta-WebIndexer is a web crawler operated by Meta. It visits websites to gather information that helps improve the quality of search results within Meta AI, including the ability to cite and link to source pages. The crawler identifies itself with the user-agent token Meta-WebIndexer. Its activity is part of Meta's broader effort to enhance how its AI surfaces and attributes web content. The crawler's posture toward robots.txt directives is currently unverified, meaning its exact compliance behavior is not publicly documented by Meta. Site owners can control its access through standard robots.txt rules.

What it's for

For a site owner, Meta-WebIndexer can influence how your content appears in Meta AI search results. Allowing the crawler may enable Meta AI to cite and link to your pages, potentially increasing referral traffic and visibility within Meta's ecosystem. Blocking it could reduce the likelihood of your content being surfaced or attributed in Meta AI search, which may limit your reach to users relying on that platform for information discovery.

Allowing Meta-WebIndexer creates the possibility of retrieval. To find out whether Meta AI actually cites or recommends you, track your Meta AI mentions over time.

How to handle Meta-WebIndexer

To manage Meta-WebIndexer, add a rule in your robots.txt file targeting the user-agent token Meta-WebIndexer. If you wish to prevent it from crawling any part of your site, use a Disallow directive. Because its robots.txt compliance is unverified, blocking may not be guaranteed, but it is the standard method to signal your preference. Monitor your server logs for actual behavior if precise control is needed.

robots.txt rule

User-agent: Meta-WebIndexer Disallow: /

Blocking cost

Blocking Meta-WebIndexer may reduce the chance that Meta AI cites or links to your content in its search results, potentially lowering your visibility and referral traffic from Meta's AI-powered features.

Examples

• A news website allows Meta-WebIndexer, and its articles may appear with direct links in Meta AI search results when users ask about current events.
• An e-commerce site blocks the crawler, and its product pages might not be cited or linked in Meta AI shopping-related queries.
• A blog permits crawling, and its how-to guides could be surfaced as source links in Meta AI answers to instructional questions.

Related bots

• Claude-SearchBot: Also tracked as a search crawler.
• PerplexityBot: Also tracked as a search crawler.
• PhindBot: Also tracked as a search crawler.
• Applebot: Also tracked as a search crawler.
• Bravebot: Also tracked as a search crawler.
• Kimi-SearchBot: Also tracked as a search crawler.
• MistralAI-Index: Also tracked as a search crawler.
• OAI-SearchBot: Also tracked as a search crawler.
• ExaSearchBot: Also tracked as a search crawler.
• Meta AI: Meta-WebIndexer connects this operator term to its crawler behavior.
• Noindex: Meta-WebIndexer gives crawler context for Noindex.