What is bedrockbot? AI crawler guide

bedrockbot by Amazon: Amazon Bedrock web crawler connector token for customer-configured AI applications. Check its reported user-agent, robots.txt behavior, source, and verification guidance.

Amazon Bedrock web crawler connector token for customer-configured AI applications.

What is bedrockbot?

bedrockbot is the web crawler user-agent token for the Amazon Bedrock web crawl data source connector. It is operated by Amazon and allows customers to configure their own AI applications to fetch web content through Bedrock. When a site sees a request from bedrockbot, it means a specific Bedrock customer has chosen that URL as a data source for their application, not that Amazon is performing a broad, general-purpose crawl. The crawler honors robots.txt directives, so site owners can control its access using standard rules.

What it's for

For a site owner, a visit from bedrockbot indicates that your content may be used as a data source in a customer-configured AI application running on Amazon Bedrock. This could mean your information is being pulled into a private, customer-specific AI workflow rather than a public search index. Understanding this helps you decide whether to allow or block the crawler based on how you want your content to be used in third-party AI applications.

Allowing bedrockbot only creates the possibility of retrieval. To find out whether your pages are selected, measure which answer engines actually cite your pages using a stable query set.

How to handle bedrockbot

To manage bedrockbot, add a rule in your robots.txt file targeting the user-agent token 'bedrockbot'. Since the crawler honors robots.txt, you can disallow specific paths or the entire site. Consider the potential benefits of being included in AI applications before blocking, as it may provide visibility or utility to users of those applications.

robots.txt rule

User-agent: bedrockbot Disallow: /

Blocking cost

Blocking bedrockbot may prevent your content from being used as a data source in Amazon Bedrock customer applications, which could reduce your visibility in AI-driven answers, citations, or agent workflows that rely on those applications.

Examples

• A Bedrock customer configures a web crawl data source for their AI application, and bedrockbot fetches pages from the specified domains.
• bedrockbot respects a robots.txt disallow rule and skips crawling restricted directories on a site.
• A site owner sees bedrockbot in their logs and knows it is tied to a specific customer's Bedrock application, not a general Amazon crawl.

Related bots

• Google-CloudVertexBot: Also tracked as a general crawler.
• Google-Firebase: Also tracked as a general crawler.
• Scrapy: Also tracked as a general crawler.
• Amazonbot: Another Amazon general crawler to compare.
• GoogleOther-Image: Also tracked as a general crawler.
• GoogleOther-Video: Also tracked as a general crawler.
• FirecrawlAgent: Also tracked as a general crawler.
• Brightbot: Also tracked as a general crawler.
• ApifyBot: Also tracked as a general crawler.
• Robots.txt: Robots.txt is the control file used to allow or block bedrockbot.
• Alexa: bedrockbot gives crawler context for Alexa.