What is SemrushBot-OCOB? AI crawler guide

SemrushBot-OCOB by Semrush: Semrush crawler for the ContentShake AI content tool. Check its reported user-agent, robots.txt behavior, source, and verification guidance.

Semrush crawler for the ContentShake AI content tool.

What is SemrushBot-OCOB?

SemrushBot-OCOB is a web crawler operated by Semrush. It is specifically designed to support the ContentShake AI content tool. The bot visits web pages to gather data that ContentShake uses to generate reports and insights. It identifies itself with the user-agent token SemrushBot-OCOB and honors robots.txt directives. This crawler is part of Semrush's suite of SEO and content tools, helping users analyze and improve their online content. Its activity is limited to the needs of the ContentShake tool, and it does not perform general web indexing. Site owners can control its access through standard robots.txt rules.

What it's for

For site owners, SemrushBot-OCOB enables Semrush's ContentShake AI tool to access and analyze their web pages. This can help in generating content reports and insights that may improve the site's content strategy. Allowing the bot ensures that ContentShake can provide accurate and up-to-date information about the site. Blocking it may prevent the tool from checking the site, potentially affecting the quality of reports and recommendations.

Allowing SemrushBot-OCOB only creates the possibility of retrieval. To find out whether your pages are selected, measure which answer engines actually cite your pages using a stable query set.

How to handle SemrushBot-OCOB

To manage SemrushBot-OCOB, use its user-agent token in your robots.txt file. Since it honors robots.txt, you can disallow it from specific paths or your entire site. The page will render the appropriate snippet separately. If you use ContentShake and want accurate reports, consider allowing the bot. If you do not use the tool or prefer to restrict access, blocking it is straightforward and will be respected.

robots.txt rule

User-agent: SemrushBot-OCOB Disallow: /

Blocking cost

Blocking SemrushBot-OCOB can prevent Semrush's ContentShake AI tool from accessing your pages, which may affect the accuracy and completeness of any content reports or insights generated for your site.

Examples

• A marketing team uses ContentShake to generate content ideas; SemrushBot-OCOB crawls their blog to analyze existing posts and suggest improvements.
• An e-commerce site allows SemrushBot-OCOB so ContentShake can review product descriptions and recommend optimizations for better engagement.
• A news website blocks SemrushBot-OCOB via robots.txt, and as a result, ContentShake cannot include their articles in its content analysis reports.

Related bots

• SemrushBot-SWA: Another Semrush SEO crawler to compare.
• Claude-User: Shares crawler-policy context with SemrushBot-OCOB.
• GoogleOther: Shares crawler-policy context with SemrushBot-OCOB.
• KimiBot: Shares crawler-policy context with SemrushBot-OCOB.
• LAIONDownloader: Shares crawler-policy context with SemrushBot-OCOB.
• ApifyBot: Shares crawler-policy context with SemrushBot-OCOB.
• Claude-SearchBot: Shares crawler-policy context with SemrushBot-OCOB.
• ClaudeBot: Shares crawler-policy context with SemrushBot-OCOB.
• Google-Extended: Shares crawler-policy context with SemrushBot-OCOB.
• Robots.txt: Robots.txt is the control file used to allow or block SemrushBot-OCOB.
• Crawling: SemrushBot-OCOB is a concrete crawler example for this concept.