What is OAI-SearchBot? AI crawler guide
OAI-SearchBot by OpenAI: OpenAI search crawler for indexing pages that can appear in ChatGPT search results. Check its reported user-agent, robots.txt behavior, source, and verification guidance.
OpenAI search crawler for indexing pages that can appear in ChatGPT search results.
What is OAI-SearchBot?
OAI-SearchBot is a web crawler operated by OpenAI. It visits pages to index content that can appear in ChatGPT search results. The bot identifies itself with the user-agent token OAI-SearchBot and respects robots.txt directives. Its purpose is to discover and catalog web pages so that ChatGPT can surface them as search results or source citations when users ask questions. The crawler does not collect data for training OpenAI's generative AI models; it is solely used for search indexing. Site owners can control its access through standard robots.txt rules.
What it's for
If you want your pages to be eligible to appear in ChatGPT search results and be cited as sources, you should allow OAI-SearchBot to crawl your site. Blocking it may prevent your content from being discovered and shown to users who rely on ChatGPT for search.
Allowing OAI-SearchBot creates the possibility of retrieval. To find out whether ChatGPT actually cites or recommends you, track your ChatGPT mentions over time.
How to handle OAI-SearchBot
To allow OAI-SearchBot, ensure your robots.txt file does not disallow it. To block it, add a disallow rule for the user-agent OAI-SearchBot. The bot honors robots.txt, so this is the primary control mechanism.
robots.txt rule
User-agent: OAI-SearchBot Disallow: /
Blocking cost
Blocking OAI-SearchBot can prevent your pages from appearing in ChatGPT search results and being eligible for source citations.
Examples
- A news website allows OAI-SearchBot so its articles can be indexed and shown as search results in ChatGPT.
- A private dashboard blocks OAI-SearchBot via robots.txt to keep its pages out of ChatGPT search.
- A blog that relies on organic traffic from AI search engines permits OAI-SearchBot to crawl its posts.
Related bots
- PerplexityBot: Also tracked as a search crawler.
- Kimi-SearchBot: Also tracked as a search crawler.
- Amzn-SearchBot: Also tracked as a search crawler.
- MistralAI-Index: Also tracked as a search crawler.
- ExaSearchBot: Also tracked as a search crawler.
- Kagibot: Also tracked as a search crawler.
- Claude-SearchBot: Also tracked as a search crawler.
- YouBot: Also tracked as a search crawler.
- Googlebot: Also tracked as a search crawler.
- SearchGPT: OAI-SearchBot is a search crawler tied to AI answer visibility.
- ChatGPT: OAI-SearchBot gives crawler context for ChatGPT.
Frequently Asked Questions
What does OAI-SearchBot do?
OAI-SearchBot crawls web pages to index them for inclusion in ChatGPT search results. It does not use crawled data for training OpenAI's generative models.
How can I block OAI-SearchBot?
You can block OAI-SearchBot by adding a disallow rule for the user-agent OAI-SearchBot in your site's robots.txt file. The bot honors robots.txt directives.
Will blocking OAI-SearchBot affect my site's visibility in ChatGPT?
Yes, blocking OAI-SearchBot may prevent your pages from appearing in ChatGPT search results and being used as source citations.
Does OAI-SearchBot use my content for AI training?
No, OAI-SearchBot is only used for search indexing. It does not collect data for training OpenAI's generative AI models.
How can I verify that OAI-SearchBot is crawling my site?
Check for the OAI-SearchBot token, then compare the source IP with OpenAI's published searchbot.json ranges. User-agent text alone can be spoofed.
Data & Sources
- OpenAI documentation - Primary source for OAI-SearchBot crawler details.
- OAI-SearchBot verification: Published IP ranges - Verification information published for OAI-SearchBot.
- OAI-SearchBot live crawler data - Trakkr crawler telemetry for this user agent.