AI crawler market share.
A live read on which AI systems are fetching the web most often, split by training crawlers, search indexing bots, conversation fetches, and agents.
Which AI crawlers fetch the web most?
41 rows · updated June 8, 2026
| Crawler | Type | Visits | Market share | Brands seen | Unique URLs |
|---|---|---|---|---|---|
| Bytespider | Training crawler | 8,030,373 | 49.0% | 82 | 0 |
| ClaudeBot | Training crawler | 1,736,590 | 10.6% | 86 | 0 |
| ChatGPT-User | Conversation fetch | 1,540,508 | 9.4% | 40 | 0 |
| Applebot | AI search index | 1,504,844 | 9.2% | 28 | 0 |
| Amazonbot | Training crawler | 1,015,896 | 6.2% | 38 | 0 |
| OAI-SearchBot | AI search index | 901,386 | 5.5% | 101 | 0 |
| GPTBot | Training crawler | 868,367 | 5.3% | 122 | 0 |
| Scrapy | Unknown | 339,806 | 2.1% | 12 | 0 |
| Meta-ExternalFetcher | Conversation fetch | 197,356 | 1.2% | 50 | 0 |
| PerplexityBot | AI search index | 140,159 | 0.9% | 36 | 0 |
| GoogleOther | Unknown | 16,606 | 0.1% | 1 | 0 |
| Claude-User | Conversation fetch | 11,841 | 0.1% | 20 | 0 |
| Meta-ExternalAgent | Unknown | 11,760 | 0.1% | 9 | 0 |
| Bingbot | Unknown | 11,375 | 0.1% | 9 | 0 |
| Claude-SearchBot | AI search index | 11,204 | 0.1% | 12 | 0 |
| CCBot | Training crawler | 11,193 | 0.1% | 13 | 0 |
| ChatGPT | Unknown | 9,388 | 0.1% | 8 | 0 |
| GrokBot | Unknown | 8,300 | 0.1% | 12 | 0 |
| YouBot | AI search index | 7,834 | 0.0% | 12 | 0 |
| Perplexity-User | Conversation fetch | 3,492 | 0.0% | 13 | 0 |
| bytespider | Training crawler | 2,748 | 0.0% | 1 | 0 |
| DeepSeekBot | Unknown | 2,051 | 0.0% | 19 | 0 |
| Claude | Unknown | 1,661 | 0.0% | 8 | 0 |
| MistralAI-User | Conversation fetch | 1,431 | 0.0% | 43 | 0 |
| Googlebot | Unknown | 911 | 0.0% | 1 | 0 |
| Perplexity | Unknown | 752 | 0.0% | 6 | 0 |
| Bing | Unknown | 682 | 0.0% | 3 | 0 |
| Amazon | Unknown | 645 | 0.0% | 3 | 0 |
| Meta AI | Unknown | 503 | 0.0% | 3 | 0 |
| cohere-ai | Unknown | 225 | 0.0% | 13 | 0 |
| Claude-Web | Unknown | 203 | 0.0% | 7 | 0 |
| Diffbot | Unknown | 177 | 0.0% | 10 | 0 |
| facebookexternalhit | Unknown | 171 | 0.0% | 1 | 0 |
| Google-Agent | Agent | 88 | 0.0% | 5 | 0 |
| AI2Bot | Unknown | 82 | 0.0% | 4 | 0 |
| anthropic-ai | Unknown | 82 | 0.0% | 6 | 0 |
| Timpibot | Unknown | 37 | 0.0% | 2 | 0 |
| MistralBot | Unknown | 21 | 0.0% | 7 | 0 |
| Mistral | Unknown | 4 | 0.0% | 1 | 0 |
| Omgili | Unknown | 3 | 0.0% | 2 | 0 |
| Google-Extended | Unknown | 2 | 0.0% | 2 | 0 |
Crawler market share
Share of observed AI crawler visits, by user agent
How this benchmark is built
Crawler visits are grouped from Trakkr's AI crawler behavior research data.
Crawler type is assigned from known bot patterns for training, indexing, conversation fetches, and agents.
Market share is the share of observed crawler visits within the current research window.
Some crawlers serve multiple product surfaces; type labels describe the primary observed role.
Related benchmarks
Published as an open dataset under CC BY 4.0. Fetch the live numbers as JSON above.
What counts as an AI crawler visit?
The crawler benchmark counts observed visits from known AI user agents such as GPTBot, OAI-SearchBot, ClaudeBot, Bytespider, PerplexityBot, and ChatGPT-User.
What is the difference between crawler types?
Training crawlers collect model-training data, search index crawlers build retrieval indexes, conversation fetches happen during user sessions, and agents act on behalf of tools or assistants.
Track your own AI visibility
Connect analytics, prompts, citations, and crawler data in Trakkr to see how your brand moves inside AI search.