What is ApifyBot? AI crawler guide

ApifyBot by Apify: Token associated with crawlers run on the Apify scraping platform by its customers. Check its reported user-agent, robots.txt behavior, source, and verification guidance.

Token associated with crawlers run on the Apify scraping platform by its customers.

What is ApifyBot?

ApifyBot is a user-agent token associated with crawlers run on the Apify scraping platform by its customers. Apify provides a cloud-based environment where users can build and deploy web scrapers and crawlers for various data extraction tasks. The ApifyBot token is one of several identifiers that may appear in requests originating from the platform. Its behavior is not centrally controlled by Apify; instead, each customer configures their crawler's settings, including how it interacts with robots.txt files. This means that compliance with robots.txt directives can vary from one crawler to another. Site owners should be aware that the presence of ApifyBot in their logs indicates traffic from an Apify-hosted crawler, but the specific purpose and behavior depend entirely on the customer operating it.

What it's for

For a site owner, ApifyBot represents traffic from the Apify platform, which could be used for anything from legitimate research and monitoring to competitive scraping or content aggregation. Because the crawler's behavior is determined by the Apify customer, the impact on your site can vary widely. Blocking ApifyBot may prevent some unwanted scraping, but it could also block benign or beneficial crawlers that respect robots.txt. Since Apify's main content crawler does not send an identifying user-agent, blocking ApifyBot alone may not stop all Apify-originated traffic.

How to handle ApifyBot

To block crawlers that identify as ApifyBot, add a robots.txt rule disallowing the user-agent ApifyBot. Keep in mind that this only affects crawlers configured to send this token and to respect robots.txt. Since compliance is per-crawler and set by the Apify customer, some may ignore the rule. Monitor your logs for other user-agents or patterns that may indicate Apify traffic. If you need to manage Apify platform access more broadly, consider additional measures such as rate limiting or analyzing request patterns, as the main content crawler may not identify itself.

robots.txt rule

User-agent: ApifyBot Disallow: /

Blocking cost

Blocking ApifyBot may prevent some Apify-hosted crawlers from accessing your site, but it is unlikely to affect visibility in AI search, answers, citations, training data, or agent interactions, as this token is not specifically tied to those functions.

Examples

• A market research firm uses an Apify crawler that sends the ApifyBot token to collect publicly available pricing data from e-commerce sites, respecting robots.txt if configured to do so.
• A developer deploys a personal project on Apify to monitor changes on a blog, with the crawler identifying as ApifyBot and following the site's crawl delay directives.
• A site owner may see the ApifyBot token in logs when Apify fetches a page for its other workflow.

Related bots

• FirecrawlAgent: Also tracked as a general crawler.
• Brightbot: Also tracked as a general crawler.
• Scrapy: Also tracked as a general crawler.
• GoogleOther: Also tracked as a general crawler.
• GoogleOther-Image: Also tracked as a general crawler.
• GoogleOther-Video: Also tracked as a general crawler.
• meta-externalads: Also tracked as a general crawler.
• OAI-AdsBot: Also tracked as a general crawler.
• bedrockbot: Also tracked as a general crawler.
• Robots.txt: Robots.txt is the control file used to allow or block ApifyBot.
• AI Crawlers: ApifyBot is a concrete crawler example for this concept.