What is ShapBot? AI crawler guide

ShapBot by Parallel: Parallel crawler for discovering and indexing websites for Parallel web APIs. Check its reported user-agent, robots.txt behavior, source, and verification guidance.

Parallel crawler for discovering and indexing websites for Parallel web APIs.

What is ShapBot?

ShapBot is a web crawler operated by Parallel that discovers and indexes websites for Parallel web APIs. It identifies itself with the user-agent token ShapBot and honors the Robots Exclusion Protocol, meaning it respects robots.txt directives. The crawler is part of Parallel's infrastructure for making web content accessible through their API services. It systematically visits pages to build an index that powers Parallel's search and retrieval capabilities, enabling developers to integrate web data into their applications. ShapBot's crawling behavior is documented by Parallel, and site owners can control its access through standard robots.txt rules.

What it's for

For site owners, ShapBot's crawling can make their content discoverable through Parallel's web APIs, potentially increasing visibility in applications that rely on Parallel's search and indexing services. If you want your site to appear in results served by Parallel-powered tools, allowing ShapBot access is beneficial. Conversely, if you prefer not to have your content indexed by Parallel, blocking ShapBot can prevent it from being included in their API responses.

Whether to allow ShapBot is a policy call. Either way, check which AI platforms actually send you human visitors before assuming crawl volume equals value.

How to handle ShapBot

To manage ShapBot's access, add directives for the user-agent ShapBot in your robots.txt file. Since ShapBot honors robots.txt, you can disallow specific paths or the entire site. The page will render a snippet showing how to block ShapBot completely. If you want to allow crawling, simply omit any disallow rules for ShapBot, or explicitly allow it. There are no additional controls needed beyond standard robots.txt configuration.

robots.txt rule

User-agent: ShapBot Disallow: /

Blocking cost

Blocking ShapBot may prevent your site from being indexed by Parallel, which could reduce its visibility in applications that use Parallel's web APIs for search and data retrieval.

Examples

• ShapBot crawls a news website, indexing articles so they can be retrieved via Parallel's search API.
• A developer uses Parallel's API to query for product information, and ShapBot has previously indexed an e-commerce site, making its product pages available in the API response.
• ShapBot respects a robots.txt disallow rule for a private section of a website, skipping those pages during its crawl.

Related bots

• Googlebot: Also tracked as a search crawler.
• MistralAI-Index: Also tracked as a search crawler.
• Bingbot: Also tracked as a search crawler.
• Bravebot: Also tracked as a search crawler.
• Cloudflare-AutoRAG: Also tracked as a search crawler.
• OAI-SearchBot: Also tracked as a search crawler.
• ExaSearchBot: Also tracked as a search crawler.
• Kagibot: Also tracked as a search crawler.
• KlaviyoAIBot: Also tracked as a search crawler.
• Robots.txt: Robots.txt is the control file used to allow or block ShapBot.
• AI Search: ShapBot is a search crawler tied to AI answer visibility.