What is Kagibot? AI crawler guide
Kagibot by Kagi: Kagi search crawler that indexes pages for its search index and the answers built on it. Check its reported user-agent, robots.txt behavior, source, and verification guidance.
Kagi search crawler that indexes pages for its search index and the answers built on it.
What is Kagibot?
Kagibot is the web crawler operated by Kagi, a privacy-focused search engine. It visits websites to collect content for Kagi's search index and the AI-powered answers built on that index. The crawler identifies itself with the user-agent token Kagibot and respects the Robots Exclusion Protocol. Site owners can control its access through robots.txt directives. The token is case-sensitive; rules written for KagiBot with a capital B will not match Kagibot. Its full user-agent string includes a reference to Kagi's bot information page.
What it's for
For site owners, Kagibot's visits mean that pages may appear in Kagi search results and its assistant answers. Allowing the crawler can increase visibility on a privacy-oriented search platform. Blocking it may remove your pages from Kagi's search index and the answers derived from it, potentially reducing traffic from users who prefer Kagi.
How to handle Kagibot
To manage Kagibot's access, add a rule in your robots.txt file targeting the user-agent token Kagibot. Use the Disallow directive to block specific paths or the entire site. Ensure the token is written exactly as Kagibot, because the crawler is case-sensitive and will not recognize variations like KagiBot. The page will render a basic block snippet separately.
robots.txt rule
User-agent: Kagibot Disallow: /
Blocking cost
Blocking Kagibot can prevent your pages from appearing in Kagi search results and its assistant answers, which may reduce visibility and traffic from Kagi users.
Examples
- Kagibot crawls a news article, and the article later appears in Kagi search results for relevant queries.
- A product page is indexed by Kagibot, and Kagi's assistant can then cite it in an answer about that product.
- A blog post disallowed in robots.txt for Kagibot is not included in Kagi's search index or assistant answers.
Related bots
- Kimi-SearchBot: Also tracked as a search crawler.
- Amzn-SearchBot: Also tracked as a search crawler.
- ExaSearchBot: Also tracked as a search crawler.
- OAI-SearchBot: Also tracked as a search crawler.
- Claude-SearchBot: Also tracked as a search crawler.
- PerplexityBot: Also tracked as a search crawler.
- YouBot: Also tracked as a search crawler.
- xAI-SearchBot: Also tracked as a search crawler.
- MistralAI-Index: Also tracked as a search crawler.
- Robots.txt: Robots.txt is the control file used to allow or block Kagibot.
- SearchGPT: Kagibot is a search crawler tied to AI answer visibility.
Frequently Asked Questions
What does Kagibot do?
Kagibot is Kagi's web crawler. It indexes pages for Kagi's search engine and the AI-powered answers built on that index.
How can I block Kagibot?
You can block Kagibot by adding a User-agent: Kagibot rule in your robots.txt file, followed by Disallow directives for the paths you want to restrict.
Is the Kagibot user-agent token case-sensitive?
Yes, the token is case-sensitive. It must be written as Kagibot. Using KagiBot or other variations will not match the crawler.
What happens if I block Kagibot?
Blocking Kagibot can remove your pages from Kagi search results and its assistant answers, potentially reducing traffic from Kagi users.
Does Kagibot honor robots.txt?
Yes, Kagibot honors the Robots Exclusion Protocol and respects directives set in robots.txt files.
Data & Sources
- Kagi documentation - Primary source for Kagibot crawler details.
- Kagibot verification: Published IPs and reverse DNS - Kagi publishes its source IPs and its crawler resolves under kagibot.org.