What is FacebookBot? AI crawler guide
FacebookBot by Meta: Meta crawler historically documented for Facebook crawling and AI-related training uses. Check its reported user-agent, robots.txt behavior, source, and verification guidance.
Meta crawler historically documented for Facebook crawling and AI-related training uses.
What is FacebookBot?
FacebookBot is Meta's web crawler, historically used for Facebook content sharing and more recently associated with AI training. It fetches public web pages to generate link previews when content is shared on Facebook and may also collect data for Meta's AI models. The crawler identifies itself with the user-agent token FacebookBot and respects robots.txt directives. It is operated by Meta and documented in Facebook's developer resources for webmasters.
What it's for
For site owners, FacebookBot enables rich link previews when pages are shared on Facebook, which can drive traffic and engagement. It may also contribute to Meta's AI training datasets, potentially influencing how content surfaces in AI-powered features. Allowing the bot supports social visibility, while blocking it could limit these benefits.
Allowing FacebookBot creates the possibility of retrieval. To find out whether Meta AI actually cites or recommends you, track your Meta AI mentions over time.
How to handle FacebookBot
To control FacebookBot, add a robots.txt rule targeting the user-agent token FacebookBot. The bot honors standard robots.txt directives, so you can disallow specific paths or the entire site. Be aware that blocking may affect how your pages appear when shared on Facebook.
robots.txt rule
User-agent: FacebookBot Disallow: /
Blocking cost
Blocking FacebookBot can prevent your pages from generating link previews on Facebook and may exclude your content from Meta's AI training data, potentially reducing visibility in AI-driven features.
Examples
- When a user shares a link on Facebook, FacebookBot crawls the page to fetch the title, description, and thumbnail image for the preview card.
- FacebookBot may crawl public web pages to collect data used in training Meta's large language models, such as those powering AI assistants.
- If a site disallows FacebookBot in robots.txt, shared links may show a generic placeholder instead of a rich preview on Facebook.
Related bots
- TikTokSpider: Also tracked as a social crawler.
- facebookexternalhit: Another Meta social crawler to compare.
- Meta-ExternalAgent: Also operated by Meta.
- Meta-ExternalFetcher: Also operated by Meta.
- Amazonbot: Shares crawler-policy context with FacebookBot.
- Applebot: Shares crawler-policy context with FacebookBot.
- GoogleOther: Shares crawler-policy context with FacebookBot.
- GPTBot: Shares crawler-policy context with FacebookBot.
- Claude-Web: Shares crawler-policy context with FacebookBot.
- Meta AI: FacebookBot connects this operator term to its crawler behavior.
- Robots.txt: Robots.txt is the control file used to allow or block FacebookBot.
Frequently Asked Questions
Does FacebookBot respect robots.txt?
Yes, FacebookBot honors robots.txt directives. You can control its access by specifying rules for the user-agent token FacebookBot.
What happens if I block FacebookBot?
Blocking FacebookBot may prevent your pages from displaying rich previews when shared on Facebook, and could exclude your content from Meta's AI training datasets.
Is FacebookBot used for AI training?
Meta has documented that FacebookBot is used for AI-related training purposes, in addition to its traditional role in Facebook content sharing.
How can I verify FacebookBot visits?
You can check your server logs for requests with the user-agent string FacebookBot. It typically originates from Meta's IP ranges.
Does FacebookBot follow crawl-delay directives?
FacebookBot's documentation does not explicitly mention support for crawl-delay, but it adheres to standard robots.txt rules including disallow directives.
Data & Sources
- Meta documentation - Primary source for FacebookBot crawler details.
- FacebookBot source reference - Source used to verify FacebookBot.