What is ExaSearchBot? AI crawler guide
ExaSearchBot by Exa: Crawler for Exa's search index, which AI products and agents query to retrieve source pages. Check its reported user-agent, robots.txt behavior, source, and verification guidance.
Crawler for Exa's search index, which AI products and agents query to retrieve source pages.
What is ExaSearchBot?
ExaSearchBot is the web crawler operated by Exa that builds the search index used by Exa's own search products and by third-party AI tools and agents. The bot fetches public web pages so that Exa can return relevant source links when an AI application queries its index. It identifies itself with the user-agent token ExaSearchBot and respects the Robots Exclusion Protocol. If no specific rule for ExaSearchBot appears in robots.txt, the crawler falls back to directives set for major search engines, and then to the wildcard rule. This fallback behavior means that site owners who block general crawlers may also block ExaSearchBot unless they add a dedicated rule.
What it's for
For a site owner, ExaSearchBot determines whether your pages appear in the search results that Exa delivers to AI products and agents. If you want your content to be discoverable by AI tools that rely on Exa's index, you should allow the bot. Conversely, if you prefer not to have your pages surfaced through Exa's ecosystem, you can block it. Because ExaSearchBot falls back to rules for major search engines when no specific rule exists, your existing robots.txt directives may already affect its access.
How to handle ExaSearchBot
To control ExaSearchBot, add a dedicated rule in your robots.txt file using the user-agent token ExaSearchBot. You can allow full access, disallow specific paths, or block the bot entirely. If you do not include a rule for ExaSearchBot, the crawler will check for directives aimed at major search engines and then the wildcard rule, which may lead to unintended crawling or blocking. Review your current robots.txt to ensure the fallback behavior aligns with your intentions.
robots.txt rule
User-agent: ExaSearchBot Disallow: /
Blocking cost
Blocking ExaSearchBot can prevent your pages from appearing in search results served to AI tools and agents that depend on Exa's index, potentially reducing your visibility in AI-driven search experiences.
Examples
- An AI assistant queries Exa's index for recent research on climate change and receives a link to a scientific paper that ExaSearchBot previously crawled.
- A developer builds a custom AI agent that uses Exa's search API to retrieve product documentation, and the agent returns a page that was indexed by ExaSearchBot.
- A news website blocks ExaSearchBot in robots.txt, and its articles no longer appear in results when an AI tool searches Exa's index for current events.
Related bots
- Kagibot: Also tracked as a search crawler.
- OAI-SearchBot: Also tracked as a search crawler.
- PerplexityBot: Also tracked as a search crawler.
- Kimi-SearchBot: Also tracked as a search crawler.
- MistralAI-Index: Also tracked as a search crawler.
- YouBot: Also tracked as a search crawler.
- Claude-SearchBot: Also tracked as a search crawler.
- xAI-SearchBot: Also tracked as a search crawler.
- Amzn-SearchBot: Also tracked as a search crawler.
- Robots.txt: Robots.txt is the control file used to allow or block ExaSearchBot.
- SearchGPT: ExaSearchBot is a search crawler tied to AI answer visibility.
Frequently Asked Questions
Does ExaSearchBot respect robots.txt?
Yes, ExaSearchBot honors the Robots Exclusion Protocol. If you set a rule for the user-agent token ExaSearchBot, the crawler will follow it. If no specific rule exists, it falls back to rules for major search engines and then to the wildcard rule.
What happens if I don't have a rule for ExaSearchBot?
If your robots.txt does not mention ExaSearchBot, the crawler will look for directives aimed at major search engines. If those are absent, it will follow the wildcard rule. This fallback can cause the bot to be blocked or allowed based on your general settings.
Can blocking ExaSearchBot affect my visibility in AI tools?
Yes, blocking ExaSearchBot may prevent your pages from being included in Exa's search index. Since many AI products and agents use Exa's index to retrieve source pages, your content could become invisible to those tools.
How can I allow ExaSearchBot to crawl my site?
Add a rule in your robots.txt file with the user-agent token ExaSearchBot and specify the paths you want to allow. For example, you can use 'Disallow:' with no path to permit crawling of your entire site.
Does ExaSearchBot follow the same rules as Googlebot?
ExaSearchBot does not automatically follow Googlebot's rules. However, if no specific rule for ExaSearchBot exists, it falls back to directives for major search engines, which may include Googlebot. For precise control, add a dedicated rule for ExaSearchBot.
Data & Sources
- Exa documentation - Primary source for ExaSearchBot crawler details.
- ExaSearchBot verification: Signed agent key directory - Verification information published for ExaSearchBot.