What is Cloudflare-AutoRAG? AI crawler guide

Cloudflare-AutoRAG by Cloudflare: Cloudflare AutoRAG crawler used to index configured content for AI search applications. Check its reported user-agent, robots.txt behavior, source, and verification guidance.

Cloudflare AutoRAG crawler used to index configured content for AI search applications.

What is Cloudflare-AutoRAG?

Cloudflare-AutoRAG is a web crawler operated by Cloudflare that indexes content for AI search applications. It is part of Cloudflare's AutoRAG service, which allows site owners to configure specific content to be included in AI-powered search experiences. The crawler visits web pages to gather data that can be used to generate answers or retrieve information in response to user queries. It identifies itself with the user-agent token Cloudflare-AutoRAG and honors robots.txt directives, meaning it respects standard exclusion rules set by website administrators. Its activity is typically seen on sites that have explicitly set up Cloudflare AutoRAG to index their content.

What it's for

For site owners, a visit from Cloudflare-AutoRAG indicates that their site or application is configured to use Cloudflare's AutoRAG service. This means their content may be indexed and made available through AI search applications powered by Cloudflare. It can help drive visibility and traffic by enabling their information to appear in AI-generated answers or search results. Since the crawler only indexes content that has been configured for AutoRAG, its presence suggests an intentional integration rather than unsolicited crawling.

Whether to allow Cloudflare-AutoRAG is a policy call. Either way, check which AI platforms actually send you human visitors before assuming crawl volume equals value.

How to handle Cloudflare-AutoRAG

If you want to prevent Cloudflare-AutoRAG from accessing your site, you can block it using your robots.txt file. The crawler honors robots.txt directives, so adding a disallow rule for the user-agent token Cloudflare-AutoRAG will stop it from crawling your pages. If you are using Cloudflare AutoRAG intentionally, ensure your robots.txt allows the crawler to access the content you want indexed. No other special handling is required.

robots.txt rule

User-agent: Cloudflare-AutoRAG Disallow: /

Blocking cost

Blocking Cloudflare-AutoRAG may prevent your content from being indexed and used in AI search applications powered by Cloudflare's AutoRAG service, potentially reducing visibility in those AI-driven experiences.

Examples

• A website configured with Cloudflare AutoRAG sees the crawler fetch its product pages to include in an AI shopping assistant.
• A documentation site allows Cloudflare-AutoRAG to index its guides, enabling users to get AI-generated answers from that content.
• A news publisher blocks Cloudflare-AutoRAG via robots.txt, so its articles are not used in Cloudflare's AI search results.

Related bots

• KlaviyoAIBot: Also tracked as a search crawler.
• amazon-kendra: Also tracked as a search crawler.
• atlassian-bot: Also tracked as a search crawler.
• Bravebot: Also tracked as a search crawler.
• Googlebot: Also tracked as a search crawler.
• MistralAI-Index: Also tracked as a search crawler.
• ShapBot: Also tracked as a search crawler.
• Bingbot: Also tracked as a search crawler.
• PhindBot: Also tracked as a search crawler.
• Robots.txt: Robots.txt is the control file used to allow or block Cloudflare-AutoRAG.
• AI Crawlers: Cloudflare-AutoRAG is a concrete crawler example for this concept.