Not sure where to start? Take a guided tour.
AI Site Grader
AI Query Scanner
llms.txt GeneratorWhat does the crawler data say about site architecture?
Mostly, AI crawlers are biased toward easy-to-hit pages and often do not traverse deeply enough to rescue weak structure. The architecture has to expose useful answers early.
The key numbers.
- One-visit sessions
- Most AI crawler visits are one-and-done sessions.
- 88.5%
- Unique URLs
- Distinct pages visited by AI crawlers.
- 314,501
- GPTBot homepage rate
- GPTBot rarely starts on the homepage.
- 2.75%
Dense internal linking and clear answer surfaces are required because bots do not spend long exploring, with 88.5% of AI crawler visits being one-and-done sessions.
The evidence.
| Measure | Result |
|---|---|
| One-visit sessions Most AI crawler visits are one-and-done sessions. | 88.5% |
| Unique URLs Distinct pages visited by AI crawlers. | 314,501 |
| GPTBot homepage rate GPTBot rarely starts on the homepage. | 2.75% |
Population and method
Built from 575,788 AI crawler visits across 84 brands and 314,501 unique URLs observed between June 11, 2025 and February 1, 2026.
Where the evidence stops
This study measures observable crawler traffic rather than hidden fetches or internal cache reads.
One dominant e-commerce brand was excluded to keep the page-type patterns more generalizable.
Visit share and homepage preference vary sharply by bot family, so averages can hide crawler-specific behavior.
What to do with this
- Optimize entry pages and internal links for the crawlers that actually show up in your logs.
- Treat crawler families differently because they do not read the same site architecture the same way.
- Use low-depth answer pages to improve the odds that a one-visit crawler still finds something citeable.
Operators must adjust site hierarchy and internal linking strategies because relying on deep crawls will fail to index critical content, directly impacting visibility in AI search features.
Questions about this answer.
Cite this answer.
Source study: Crawler Behavior“Mostly, AI crawlers are biased toward easy-to-hit pages and often do not traverse deeply enough to rescue weak structure. The architecture has to expose useful answers early.”
Mack Grenfell · What does the crawler data say about site architecture?
When AI Comes to Your Website · Trakkr Research · Updated 1 February 2026