AI crawler bot directory

Allen Institute for AI crawler used to find web content for open language model datasets.

AI2 crawler token associated with Dolma/open language model dataset collection.

Legacy Anthropic robots.txt token that predates the current ClaudeBot, Claude-User and Claude-SearchBot names.

Robots.txt control token for whether Applebot-crawled content may be used to train Apple foundation models.

ByteDance crawler associated with training and powering AI products.

Common Crawl's crawler for building public web crawl datasets used by researchers and AI builders.

Anthropic crawler for public web content that could contribute to Claude model training.

Cohere training-data crawler token reported for downloading web data for enterprise language models.

DeepSeek crawler token reported for training language models and improving AI products.

Robots.txt product token that controls eligible use of Google-crawled content for Gemini training and grounding.

OpenAI crawler for content that may be used to improve generative AI foundation models.

xAI crawler token listed by public crawler directories for Grok-related crawling.

NICT crawler for data used in artificial intelligence technologies and third-party research/commercial uses.

Open-source image dataset downloader token used to collect images for machine learning datasets.

Moonshot AI crawler for content that may be used to improve its Kimi models.

LAION downloader token used in machine learning research dataset collection.

Meta crawler for indexing content directly for AI model training and product improvement use cases.

Mistral crawler for content that may be used to train its models, kept separate from its search index crawler.

Huawei crawler token reported for training data collection for the PanGu multimodal LLM.

SB Intuitions crawler for data used in AI development and information analysis.

Ceramic AI crawler token for downloading data used to train LLMs.

Velen crawler for business datasets and machine learning models.

Webz.io token covering whether crawled content may be included in the datasets it resells for AI and machine learning use.

Amazon Kendra crawler token for intelligent enterprise search over configured content sources.

Amazon search crawler that indexes pages so they can be retrieved and cited in Amazon search and assistant answers.

Apple crawler for search experiences across Spotlight, Siri, Safari, and related Apple surfaces.

Atlassian Rovo crawler used to index connected website content for AI search, assistants, and agents.

Microsoft Bing crawler used to crawl and index pages for Bing and Microsoft search-powered experiences.

Brave Search crawler used to discover and index pages, including AI-search-adjacent retrieval.

Anthropic search crawler that indexes content to improve Claude search result relevance and accuracy.

Cloudflare AutoRAG crawler used to index configured content for AI search applications.

Crawler for Exa's search index, which AI products and agents query to retrieve source pages.

Google Search crawler used to discover, crawl, render, and index pages for Google Search.

iAsk crawler used to provide answers to user queries.

Ibou crawler for building a graph representation of the web used in search.

Kagi search crawler that indexes pages for its search index and the answers built on it.

Moonshot AI search crawler that indexes pages so they can be retrieved and cited in Kimi answers.

Klaviyo AI crawler for indexing configured content to tailor AI experiences and recommendations.

Meta crawler for improving Meta AI search result quality and source linking.

Mistral automated crawler for indexing content used by Mistral AI search in Vibe.

OpenAI search crawler for indexing pages that can appear in ChatGPT search results.

Perplexity crawler for surfacing and linking websites in Perplexity search results.

Huawei crawler used for recommendations, assistant features, and AI search services.

Phind crawler token associated with AI-enhanced developer search.

Parallel crawler for discovering and indexing websites for Parallel web APIs.

Timpi crawler reported for scraping data used in search and AI model training contexts.

Crawler token observed retrieving pages for xAI's Grok search and answer features.

Yandex crawler token for data used in YandexGPT quick answers and additional analysis.

Yandex additional crawler token for YandexGPT-related answer and analysis features.

You.com crawler for web search and AI answer experiences.

Amazon agent token reported for Buy for Me shopping actions directed by customers.

Amazon fetcher that retrieves a specific page because a person asked an Amazon assistant about it.

OpenAI agent used when ChatGPT navigates websites for user-directed tasks.

User-triggered OpenAI fetcher for ChatGPT and Custom GPT actions.

Claude Code related agent token seen in crawler/user-agent lists.

Anthropic user-triggered fetcher for Claude answers that need a specific web page.

Reported Anthropic-related token seen in public crawler registries and Trakkr detection, but absent from Anthropic's current bot documentation.

Cohere token reported for retrieving data in response to user-initiated prompts.

DuckDuckGo AI assistant fetcher used by DuckAssist to retrieve content for real-time answers.

Gemini Deep Research agent token reported for collecting and scanning resources used in research answers.

Google agent token listed by Cloudflare for AI assistant activity.

Gemini CLI related token listed in AI crawler registries for coding-agent activity.

Google fetcher that reads a URL because someone added it as a source inside Gemini or NotebookLM.

Superseded name for the Google fetcher that reads a URL when someone adds it as a source in Gemini or NotebookLM.

Google AI agent token associated with browser-style task execution.

Moonshot AI fetcher that retrieves a specific page because a person asked Kimi about it.

Meta user-requested fetcher for AI and link features across Meta products.

Mistral user-action fetcher for Vibe responses that need a source page.

Amazon Nova Act agent token reported for browser-style task execution.

User-triggered Perplexity fetcher for pages needed to answer a specific question.

Semrush crawler for the ContentShake AI content tool.

Semrush crawler for SEO Writing Assistant URL checks.

Meta crawler historically documented for Facebook crawling and AI-related training uses.

Meta link preview crawler used when content is shared on Meta family apps.

ByteDance/TikTok crawler token reported alongside AI crawling lists.

Amazon crawler used to improve its products and services and potentially train Amazon AI models.

Token associated with crawlers run on the Apify scraping platform by its customers.

Amazon Bedrock web crawler connector token for customer-configured AI applications.

Bright Data's declared crawler for collecting public web data for its own datasets.

Diffbot crawler for extracting structured web data and maintaining its knowledge graph.

Firecrawl agent token for AI scraping and web-to-LLM data extraction workflows.

Google crawler used for site-owner-requested crawls related to Vertex AI Agents.

Google Firebase AI product token reported for app-related fetches.

Google generic crawler used by product teams for publicly accessible content fetches outside core Googlebot.

Google product-specific image crawler token for publicly accessible content fetches.

Google product-specific video crawler token for public content fetches.

ImageSift crawler for public image and page data used in web intelligence products.

Meta crawler documented alongside its other external agents, used for advertising related page reads.

OpenAI crawler that reviews the safety and relevance of pages submitted as ChatGPT ad landing pages.

Webz.io crawler for collecting web data sold through APIs and datasets.

Legacy Omgili/Webz.io crawler token for web data collection.

Panscient crawler for collecting and structuring business data with AI and machine learning.

Scrapy framework user-agent commonly used for web scraping, including AI and machine learning data extraction.