What is Applebot-Extended? AI crawler guide
Applebot-Extended by Apple: Robots.txt control token for whether Applebot-crawled content may be used to train Apple foundation models. Check its robots.txt control, source, and handling guidance.
Robots.txt control token for whether Applebot-crawled content may be used to train Apple foundation models.
What is Applebot-Extended?
Applebot-Extended is a robots.txt control token provided by Apple. It does not fetch pages itself. Instead, it acts as a signal that tells Apple whether content crawled by Applebot may be used to train Apple foundation models. When a site owner disallows Applebot-Extended in robots.txt, they opt out of having their content used for Apple's generative AI training. The token is documented by Apple as a usage-control mechanism, not a crawler. Any request that carries Applebot-Extended as a fetching user-agent is not from Apple and should be treated as unverified.
What it's for
For site owners, Applebot-Extended provides a way to control whether their content can be used to train Apple's foundation models. Disallowing it means pages can still appear in Apple search results, but they will not be used for generative AI training. This allows publishers to participate in Apple's search ecosystem while withholding their content from AI model development if they choose.
Applebot-Extended collects pages for model training. Training inclusion is not the same as being cited, so measure where AI answers actually cite your site before drawing conclusions from crawl logs.
How to handle Applebot-Extended
To opt out of Apple's generative AI training, add a disallow rule for the Applebot-Extended user-agent token in your robots.txt file. This does not affect Applebot's normal crawling for search indexing. Since Applebot-Extended does not fetch pages, any request claiming to be from this user-agent is not from Apple and can be blocked or ignored at the server level.
robots.txt rule
User-agent: Applebot-Extended Disallow: /
Blocking cost
Blocking Applebot-Extended may prevent your content from being used to train Apple foundation models, but it does not affect visibility in Apple search results.
Examples
- A news website adds a disallow rule for Applebot-Extended to prevent its articles from being used in Apple's AI training while remaining discoverable in Apple search.
- A blog owner checks their server logs and sees requests with the Applebot-Extended user-agent; since Apple states this token does not fetch pages, the owner treats these requests as unverified and blocks them.
- An e-commerce site allows Applebot-Extended because it wants its product descriptions to potentially improve Apple's AI models, understanding that this does not impact search rankings.
Related bots
- GPTBot: Also tracked as a training crawler.
- Google-Extended: Also tracked as a training crawler.
- KimiBot: Also tracked as a training crawler.
- Meta-ExternalAgent: Also tracked as a training crawler.
- MistralAI-Training: Also tracked as a training crawler.
- Webzio-Extended: Also tracked as a training crawler.
- anthropic-ai: Also tracked as a training crawler.
- img2dataset: Also tracked as a training crawler.
- LAIONDownloader: Also tracked as a training crawler.
- AI Training Opt-Out: Applebot-Extended is a training crawler tied to this policy decision.
- Robots.txt: Robots.txt is the control file used to allow or block Applebot-Extended.
- GrokBot: Also tracked as a training crawler.
Frequently Asked Questions
Does Applebot-Extended crawl my site?
No. Apple documents Applebot-Extended as a usage-control token only. It does not fetch pages. Any request with this user-agent is not from Apple.
Will blocking Applebot-Extended affect my search rankings?
No. Disallowing Applebot-Extended only opts your content out of Apple's generative AI training. It does not impact how your pages appear in Apple search results.
How do I opt out of Apple's AI training?
Add a disallow rule for the Applebot-Extended user-agent token in your robots.txt file. This tells Apple not to use your content for training foundation models.
What should I do if I see requests from Applebot-Extended in my logs?
Treat them as unverified. Applebot-Extended does not fetch pages, so any request with this user-agent is not from Apple's network. You can safely block or ignore such requests.
Can I allow Applebot but block Applebot-Extended?
Yes. You can allow Applebot for search indexing while disallowing Applebot-Extended to prevent your content from being used in AI training. The two tokens are independent.
Data & Sources
- Apple documentation - Primary source for Applebot-Extended crawler details.
- Applebot-Extended verification: Applebot published IP ranges - Genuine Apple crawling comes from Apple's own network and resolves under applebot.apple.com. Requests carrying Applebot-Extended as a fetching user-agent do not.