Google-Extended robots.txt control: what it covers
Google-Extended by Google: Robots.txt product token that controls eligible use of Google-crawled content for Gemini training and grounding. Check its robots.txt control, source, and handling guidance.
Robots.txt product token that controls eligible use of Google-crawled content for Gemini training and grounding.
What is Google-Extended?
Google-Extended is a standalone product token for robots.txt. It has no separate HTTP user-agent string and does not fetch pages itself. Google applies the token to content fetched by Google crawlers when deciding whether that content is eligible for covered Gemini uses. Those uses currently include future Gemini model training, grounding in Gemini Apps, and grounding in Google Search on Vertex AI. Google states that the token does not affect a site's inclusion or ranking in Google Search.
What it's for
Use Google-Extended when your policy for Gemini training and grounding differs from your policy for Google Search crawling. Allowing Googlebot while disallowing Google-Extended keeps the page eligible for Search while opting it out of the generative AI uses covered by this token. It is not a broad control for every Google AI feature, every Vertex AI crawler, or every use of content already processed by another system.
Google-Extended controls AI training use, not whether Google Search can index you. If the practical question is whether Google’s assistant recommends your brand, monitor brand mentions in Gemini separately.
How to handle Google-Extended
Choose the policy first, then publish an explicit Google-Extended group in the robots.txt file at your domain root. Use Disallow: / for a site-wide opt-out or leave Disallow empty to allow the covered uses. Do not look for a Google-Extended visit in access logs because the token does not make its own requests. Check Google's operator documentation again before treating the rule as a permanent policy because the covered products can change.
robots.txt rule
User-agent: Google-Extended Disallow: /
Blocking cost
A disallow rule removes your content from the Gemini training and grounding uses Google lists for this token. Google says the choice has no effect on Search inclusion or ranking. The trade-off is policy control versus eligibility for those covered Gemini uses, not crawl-budget savings or a guaranteed change in AI citations.
Examples
- A publisher allows Googlebot but disallows Google-Extended because its Search and Gemini-use policies differ.
- A documentation site explicitly allows Google-Extended after confirming that covered Gemini grounding is consistent with its distribution policy.
- A site owner checks access logs for Googlebot requests, not Google-Extended requests, because Google-Extended is a product token rather than a fetching crawler.
Related bots
- Applebot-Extended: Also tracked as a training crawler.
- GPTBot: Also tracked as a training crawler.
- Meta-ExternalAgent: Also tracked as a training crawler.
- CCBot: Also tracked as a training crawler.
- img2dataset: Also tracked as a training crawler.
- LAIONDownloader: Also tracked as a training crawler.
- cohere-training-data-crawler: Also tracked as a training crawler.
- PanguBot: Also tracked as a training crawler.
- AI2Bot: Also tracked as a training crawler.
- Gemini 2.0: Google-Extended gives crawler context for Gemini 2.0.
- AI Crawlers: Google-Extended is a concrete crawler example for this concept.
Frequently Asked Questions
Does blocking Google-Extended affect my Google Search rankings?
No. Google's current documentation says Google-Extended does not affect inclusion or ranking in Google Search. It controls the Gemini training and grounding uses listed for the product token.
Is Google-Extended a crawler that visits my site?
No. It is a robots.txt product token with no separate HTTP user-agent string. Google crawlers fetch the content, then Google applies the Google-Extended rule to the covered uses.
How do I opt out with Google-Extended?
Add User-agent: Google-Extended followed by Disallow: / to the robots.txt file at your domain root. This opts the site out of the Gemini training and grounding uses Google currently lists for the token.
Can I use Google-Extended to block all Google crawlers?
No. Google-Extended does not fetch pages. Manage Search crawling with Googlebot rules and check Google's separate crawler documentation for other product-specific agents.
What happens if I allow Google-Extended?
Your Google-crawled content stays eligible for the Gemini training and grounding uses Google currently lists for this token. Allowing it does not guarantee that Google will use, cite, or surface the content.
Data & Sources
- Google documentation - Primary source for Google-Extended crawler details.