What is amazon-kendra? AI crawler guide
amazon-kendra by Amazon: Amazon Kendra crawler token for intelligent enterprise search over configured content sources. Check its reported user-agent, robots.txt behavior, source, and verification guidance.
Amazon Kendra crawler token for intelligent enterprise search over configured content sources.
What is amazon-kendra?
Amazon Kendra is an intelligent enterprise search service from Amazon. The crawler identified by the user-agent token amazon-kendra fetches content from web data sources that an organization has explicitly configured within its Kendra index. It is not a general-purpose web crawler; it only visits pages that are part of a defined data source connector. The crawler honors robots.txt directives, so site owners can control its access through standard exclusion rules. Its purpose is to ingest documents and web pages into a Kendra index, enabling employees to search internal and external content from a single interface.
What it's for
If you see hits from amazon-kendra, it likely means your site has been added as a data source in an organization's Amazon Kendra index. This is not public search engine crawling; it is a configured enterprise search connector. The crawler will fetch only the pages specified in that connector's configuration, and it respects robots.txt. For site owners, this can mean your content is being made searchable within a private enterprise environment, which may be intentional if you have a business relationship with that organization.
Whether to allow amazon-kendra is a policy call. Either way, check which AI platforms actually send you human visitors before assuming crawl volume equals value.
How to handle amazon-kendra
To prevent amazon-kendra from crawling your site, add a robots.txt rule that disallows its user-agent token. Because the crawler honors robots.txt, a simple 'Disallow: /' directive will stop all fetching. If you only want to block specific paths, you can target those directories.
robots.txt rule
User-agent: amazon-kendra Disallow: /
Blocking cost
Blocking amazon-kendra may prevent your content from appearing in an organization's internal enterprise search results, which could reduce visibility within that specific business context.
Examples
- A company configures its public documentation site as a web data source in Amazon Kendra. The amazon-kendra crawler fetches the specified pages and indexes them, allowing employees to search the documentation alongside internal wikis.
- An organization adds a partner's product catalog website to its Kendra index. The crawler visits only the URLs defined in the connector, respecting any robots.txt disallow rules set by the partner.
- A site owner notices amazon-kendra in their logs and adds a 'Disallow: /private/' rule in robots.txt. The crawler subsequently skips that directory while continuing to index allowed sections.
Related bots
- KlaviyoAIBot: Also tracked as a search crawler.
- Cloudflare-AutoRAG: Also tracked as a search crawler.
- atlassian-bot: Also tracked as a search crawler.
- Amzn-SearchBot: Another Amazon search crawler to compare.
- Kimi-SearchBot: Also tracked as a search crawler.
- YouBot: Also tracked as a search crawler.
- PhindBot: Also tracked as a search crawler.
- Meta-WebIndexer: Also tracked as a search crawler.
- PetalBot: Also tracked as a search crawler.
- Robots.txt: Robots.txt is the control file used to allow or block amazon-kendra.
- Alexa: amazon-kendra gives crawler context for Alexa.
Frequently Asked Questions
Is amazon-kendra a public search engine crawler?
No, it is not a public search engine crawler. It is the crawler for Amazon Kendra, an enterprise search service that indexes content from data sources configured by an organization. It does not discover or crawl the open web independently.
Does amazon-kendra obey robots.txt?
Yes, amazon-kendra honors robots.txt directives. You can control its access to your site using standard robots.txt rules, such as disallowing its user-agent token entirely or for specific paths.
How can I tell if amazon-kendra is crawling my site?
You can identify it by the user-agent token 'amazon-kendra' in your server logs. If you see this token, it indicates that an organization has configured your site as a data source in their Amazon Kendra index.
Will blocking amazon-kendra affect my site's visibility in public search engines?
No, blocking amazon-kendra only affects its ability to index your content for the specific enterprise Kendra instance that configured your site. It has no impact on public search engines like Google or Bing.
Data & Sources
- Amazon documentation - Primary source for amazon-kendra crawler details.
- amazon-kendra source reference - Source used to verify amazon-kendra.