What is xAI-SearchBot? AI crawler guide
xAI-SearchBot by xAI: Crawler token observed retrieving pages for xAI's Grok search and answer features. Check its reported user-agent, robots.txt behavior, source, and verification guidance.
Crawler token observed retrieving pages for xAI's Grok search and answer features.
What is xAI-SearchBot?
xAI-SearchBot is a web crawler operated by xAI. It retrieves pages to support Grok search and answer features. The user-agent token is xAI-SearchBot. xAI has not published official crawler documentation, so the token is recorded from observed requests rather than an operator page. The full user-agent string includes a reference to x.ai. The crawler's robots.txt posture is unverified, meaning its compliance with standard exclusion rules is not confirmed by the operator.
What it's for
If you allow xAI-SearchBot, your pages may be retrieved and cited in Grok answers. Blocking it could reduce your visibility in those results.
How to handle xAI-SearchBot
To prevent xAI-SearchBot from accessing your site, add a Disallow rule for the user-agent token xAI-SearchBot in your robots.txt file. Because the crawler's posture is unverified, there is no guarantee it will honor the rule.
robots.txt rule
User-agent: xAI-SearchBot Disallow: /
Blocking cost
Blocking xAI-SearchBot may reduce whether your pages can be retrieved and cited in Grok answers.
Examples
- A news article page is fetched by xAI-SearchBot and later appears as a cited source in a Grok answer.
- A product page is crawled by xAI-SearchBot, and its content may be used to inform Grok's response to a shopping query.
- A technical documentation page is accessed by xAI-SearchBot, potentially providing information for a Grok answer about that topic.
Related bots
- Kimi-SearchBot: Also tracked as a search crawler.
- Amzn-SearchBot: Also tracked as a search crawler.
- Kagibot: Also tracked as a search crawler.
- YandexAdditional: Also tracked as a search crawler.
- MistralAI-Index: Also tracked as a search crawler.
- OAI-SearchBot: Also tracked as a search crawler.
- Bravebot: Also tracked as a search crawler.
- Claude-SearchBot: Also tracked as a search crawler.
- ExaSearchBot: Also tracked as a search crawler.
- Robots.txt: Robots.txt is the control file used to allow or block xAI-SearchBot.
- AI Crawlers: xAI-SearchBot is a concrete crawler example for this concept.
- YandexAdditionalBot: Also tracked as a search crawler.
Frequently Asked Questions
What does xAI-SearchBot do?
xAI-SearchBot retrieves web pages to support xAI's Grok search and answer features.
Who operates xAI-SearchBot?
xAI-SearchBot is operated by xAI, whose domain is x.ai.
Does xAI-SearchBot obey robots.txt?
The robots.txt posture of xAI-SearchBot is unverified, so it is not confirmed whether it honors standard exclusion rules.
How can I block xAI-SearchBot?
You can block xAI-SearchBot by adding a Disallow rule for the user-agent token xAI-SearchBot in your robots.txt file, but compliance is not guaranteed.
What happens if I block xAI-SearchBot?
Blocking xAI-SearchBot may reduce whether your pages can be retrieved and cited in Grok answers.
Data & Sources
- xAI-SearchBot source reference - Source used to verify xAI-SearchBot.