What is cohere-ai? AI crawler guide
cohere-ai by Cohere: Cohere token reported for retrieving data in response to user-initiated prompts. Check its reported user-agent, robots.txt behavior, source, and verification guidance.
Cohere token reported for retrieving data in response to user-initiated prompts.
What is cohere-ai?
cohere-ai is a web crawler operated by Cohere. It retrieves data in response to user-initiated prompts for Cohere-powered products. The crawler identifies itself with the user-agent token cohere-ai. It honors robots.txt directives, so site owners can control its access through standard exclusion rules. The crawler's behavior is documented in a community-maintained repository of AI bot metrics, though no official documentation from Cohere is available. Its primary function is to fetch web content on demand when a user's prompt requires real-time information, rather than performing broad, untargeted crawling.
What it's for
For site owners, visits from cohere-ai indicate that your content may be retrieved and used in responses generated by Cohere's AI products. This can drive visibility and traffic if your content is surfaced in answer to user queries. However, it also means your content could be used to fulfill information requests without users visiting your site directly. Monitoring its activity can help you understand how your content is being accessed by AI systems.
Allowing cohere-ai only creates the possibility of retrieval. To find out whether your pages are selected, measure which answer engines actually cite your pages using a stable query set.
How to handle cohere-ai
To block cohere-ai from accessing your site, add a disallow rule in your robots.txt file targeting the user-agent token cohere-ai. Since the crawler honors robots.txt, this will prevent it from fetching your content. If you wish to allow access but limit it to certain areas, you can specify allowed and disallowed paths accordingly. Regularly review your server logs for cohere-ai activity to ensure it respects your directives and to assess its crawl frequency.
robots.txt rule
User-agent: cohere-ai Disallow: /
Blocking cost
Blocking cohere-ai may prevent your content from appearing in responses generated by Cohere's AI products, potentially reducing visibility and referral traffic from those services.
Examples
- A user asks a Cohere-powered assistant for current weather conditions, and cohere-ai fetches the relevant page from a weather site to provide the answer.
- During a conversation about recent news, cohere-ai retrieves an article from a news website to include in the response.
- A developer uses a Cohere API to answer a technical question, and cohere-ai crawls a documentation page to gather the necessary information.
Related bots
- Google-Agent: Also tracked as a live fetcher.
- Claude-Code: Also tracked as a live fetcher.
- Google-Gemini-CLI: Also tracked as a live fetcher.
- Meta-ExternalFetcher: Also tracked as a live fetcher.
- Claude-User: Also tracked as a live fetcher.
- AmazonBuyForMe: Also tracked as a live fetcher.
- Claude-Web: Also tracked as a live fetcher.
- NovaAct: Also tracked as a live fetcher.
- Google-GeminiNotebook: Also tracked as a live fetcher.
- Cohere: cohere-ai connects this operator term to its crawler behavior.
- Robots.txt: Robots.txt is the control file used to allow or block cohere-ai.
Frequently Asked Questions
What does the cohere-ai crawler do?
The cohere-ai crawler retrieves web data in response to user-initiated prompts for Cohere-powered products. It fetches content on demand rather than performing broad, untargeted crawling.
Does cohere-ai respect robots.txt?
Yes, cohere-ai honors robots.txt directives. You can control its access to your site by adding rules for the user-agent token cohere-ai.
How can I block cohere-ai from my site?
Add a disallow rule for the user-agent cohere-ai in your robots.txt file. For example, 'User-agent: cohere-ai' followed by 'Disallow: /' will block it from your entire site.
Will blocking cohere-ai affect my site's visibility in AI-generated responses?
Blocking cohere-ai may prevent your content from being used in responses generated by Cohere's AI products, which could reduce visibility and referral traffic from those services.
Is there official documentation for the cohere-ai crawler?
No official documentation from Cohere is available. Information about the crawler comes from a community-maintained repository of AI bot metrics.
Data & Sources
- cohere-ai source reference - Source used to verify cohere-ai.
- cohere-ai live crawler data - Trakkr crawler telemetry for this user agent.