Not sure where to start? Take a guided tour.
AI Site Grader
AI Query Scanner
llms.txt GeneratorThe study tracks more than 300,000 unique URLs
The crawler behavior study covers 314,501 unique URLs.
The dataset is wide enough to capture real site-architecture differences.
The numbers behind it.
- Unique URLs
- Distinct pages visited by AI crawlers.
- 314,501
- Brands analyzed
- Study-wide brand count after exclusions.
- 84
The evidence.
| Measure | Result |
|---|---|
| Unique URLs Distinct pages visited by AI crawlers. | 314,501 |
| Brands analyzed Study-wide brand count after exclusions. | 84 |
Population and method
Built from 575,788 AI crawler visits across 84 brands and 314,501 unique URLs observed between June 11, 2025 and February 1, 2026.
Where the evidence stops
This study measures observable crawler traffic rather than hidden fetches or internal cache reads.
One dominant e-commerce brand was excluded to keep the page-type patterns more generalizable.
Visit share and homepage preference vary sharply by bot family, so averages can hide crawler-specific behavior.
What to do with this
These patterns are not being inferred from a tiny slice of crawler behavior.
Cite this fact.
Source study: Crawler Behavior“The crawler behavior study covers 314,501 unique URLs.”
When AI Comes to Your Website - Trakkr Research · The study tracks more than 300,000 unique URLs
When AI Comes to Your Website · Trakkr Research · Updated 1 February 2026