What is Google-CloudVertexBot? AI crawler guide

Google-CloudVertexBot by Google: Google crawler used for site-owner-requested crawls related to Vertex AI Agents. Check its reported user-agent, robots.txt behavior, source, and verification guidance.

Google crawler used for site-owner-requested crawls related to Vertex AI Agents.

What is Google-CloudVertexBot?

Google-CloudVertexBot is a crawler operated by Google for site-owner-requested crawls related to Vertex AI Agents. It is not a general search crawler; instead, it fetches content when a site owner configures a Vertex AI workflow that requires data from their own site. The bot identifies itself with the user-agent token Google-CloudVertexBot and honors robots.txt directives. Its activity is tied to specific, owner-initiated AI agent tasks rather than broad indexing or public search crawling.

What it's for

For site owners, a hit from Google-CloudVertexBot typically means a Vertex AI Agent workflow you configured is fetching content from your site. This is not part of Google's public search index; it is a direct, owner-requested action. The bot's visits are intentional and tied to your own AI agent setups, so they can be expected when you use Vertex AI Agents that need to access your web content.

Google-CloudVertexBot access decides whether Google can fetch your pages. Whether those pages get used in generated answers is a separate question, so track your presence in Google's AI results rather than treating crawl volume as visibility.

How to handle Google-CloudVertexBot

You can control Google-CloudVertexBot using robots.txt. Since it honors robots.txt, you can disallow it from specific paths or your entire site if you do not want it to fetch content for your Vertex AI Agent workflows. Keep in mind that blocking it may prevent your own configured agents from accessing needed data.

robots.txt rule

User-agent: Google-CloudVertexBot Disallow: /

Blocking cost

Blocking Google-CloudVertexBot may prevent your own Vertex AI Agent workflows from fetching necessary content from your site, potentially breaking agent functionality.

Examples

• A site owner sets up a Vertex AI Agent to answer customer questions using their own FAQ pages; Google-CloudVertexBot fetches those pages when the agent is triggered.
• During testing of a new Vertex AI Agent, the bot crawls specified product pages to provide the agent with up-to-date information.
• A developer configures a Vertex AI Agent to monitor a status page; the bot periodically fetches that page to feed the agent.

Related bots

• Google-Firebase: Another Google general crawler to compare.
• bedrockbot: Also tracked as a general crawler.
• meta-externalads: Also tracked as a general crawler.
• GoogleOther: Another Google general crawler to compare.
• GoogleOther-Video: Another Google general crawler to compare.
• Scrapy: Also tracked as a general crawler.
• GoogleOther-Image: Another Google general crawler to compare.
• Amazonbot: Also tracked as a general crawler.
• ImagesiftBot: Also tracked as a general crawler.
• Gemini 2.0: Google-CloudVertexBot gives crawler context for Gemini 2.0.
• Gemini: Google-CloudVertexBot gives crawler context for Gemini.