What is GrokBot? AI crawler guide
GrokBot by xAI: xAI crawler token listed by public crawler directories for Grok-related crawling. Check its reported user-agent, robots.txt behavior, source, and verification guidance.
xAI crawler token listed by public crawler directories for Grok-related crawling.
What is GrokBot?
GrokBot is a crawler user-agent token associated with xAI, the company behind the Grok AI assistant. It appears in public crawler directories as a token for Grok-related crawling, but xAI has not published any official documentation for it. Across monitored sites, this token is rarely seen in logs, while the xAI-SearchBot token appears steadily. The full user-agent string is simply 'GrokBot'. Because xAI provides no guidance, the token's exact behavior and purpose remain unverified.
What it's for
If you see GrokBot in your server logs, it may indicate crawling activity related to xAI's Grok AI systems. However, since xAI does not document this crawler and it is rarely observed, its presence could be incidental or misconfigured. The more commonly seen xAI crawler is xAI-SearchBot, which is the one likely performing actual data collection for search or AI training. Site owners should focus their attention on xAI-SearchBot for any access policy decisions.
Allowing GrokBot creates the possibility of retrieval. To find out whether Grok actually cites or recommends you, track your Grok mentions over time.
How to handle GrokBot
You can block GrokBot using a robots.txt rule, but because the token is rarely seen and its behavior is unverified, blocking it is unlikely to have a noticeable effect on your site's traffic or data usage. If you want to manage xAI's crawling activity, you should instead target the xAI-SearchBot token, which is the active crawler observed in practice. Monitor your logs for xAI-SearchBot and apply your robots.txt directives there.
robots.txt rule
User-agent: GrokBot Disallow: /
Blocking cost
Blocking GrokBot is unlikely to affect your site's visibility in AI search, answers, citations, training data, or agent interactions, because the token is rarely used and xAI's actual crawling appears to be done by xAI-SearchBot.
Examples
- A site owner checks their access logs and finds a single request from GrokBot over several months, while xAI-SearchBot requests appear daily.
- A webmaster adds a robots.txt disallow rule for GrokBot and sees no change in traffic or indexing patterns from xAI services.
- An administrator reviews crawler directory listings and notices GrokBot listed, but upon investigation finds no corresponding documentation or consistent activity from that user-agent.
Related bots
- cohere-training-data-crawler: Also tracked as a training crawler.
- Applebot-Extended: Also tracked as a training crawler.
- CCBot: Also tracked as a training crawler.
- Meta-ExternalAgent: Also tracked as a training crawler.
- PanguBot: Also tracked as a training crawler.
- Ai2Bot-Dolma: Also tracked as a training crawler.
- Google-Extended: Also tracked as a training crawler.
- GPTBot: Also tracked as a training crawler.
- KimiBot: Also tracked as a training crawler.
- Robots.txt: Robots.txt is the control file used to allow or block GrokBot.
- AI Training Opt-Out: GrokBot is a training crawler tied to this policy decision.
Frequently Asked Questions
Is GrokBot the main xAI crawler?
No. xAI does not document GrokBot, and it is rarely seen in logs. The crawler that appears steadily is xAI-SearchBot, which is likely the primary crawler for xAI's data collection.
Should I block GrokBot in robots.txt?
You can, but it may have no practical effect because the token is seldom used. If you want to control xAI crawling, focus on xAI-SearchBot instead.
Does GrokBot respect robots.txt?
Since xAI provides no documentation and the token is rarely observed, its compliance with robots.txt is unverified. There is no confirmed behavior to rely on.
Will blocking GrokBot affect my site's presence in Grok AI answers?
It is unlikely, because GrokBot is not the active crawler. xAI-SearchBot is the token that appears to handle crawling, so blocking GrokBot alone probably will not change how your content appears in Grok.
Where can I find official information about GrokBot?
xAI has not published any official documentation for GrokBot. The token is listed in some public crawler directories, but no authoritative source describes its behavior or purpose.
Data & Sources
- GrokBot source reference - Source used to verify GrokBot.