AI Crawlers

AI Crawlers shows you exactly when and how AI platforms crawl your website. When GPTBot visits your pricing page at 3am, you'll know. This is ground-truth visibility into the AI learning process.

Why crawler visibility matters

AI models learn from your website in two ways:

1. 1Training data collection - Crawlers like GPTBot gather content for model training
2. 2Real-time retrieval - Crawlers like ChatGPT-User fetch pages when users ask questions

If AI crawlers aren't visiting your site, AI won't have fresh information about you. If they're only visiting certain pages, those are the pages AI "knows" about.

How it works

Trakkr uses AI Pages' platform integration to detect AI crawler visits. When an AI crawler hits your site, we capture:

This requires AI Pages to be set up on your site.

Supported crawlers

We detect and identify all major AI crawlers:

Each crawler is identified by its user agent string and displays with its platform's branding.

The dashboard

Overview metrics

Timeline chart

A stacked area chart showing crawler visits over time, color-coded by platform. This reveals:

• Crawl patterns - Do visits spike on certain days?
• Platform distribution - Which AI platforms are most active?
• Trends - Is AI crawling your site more or less over time?

Top pages

Which pages do AI crawlers visit most? This list shows:

• Page URL
• Total crawl count
• Which crawlers visited
• Last visit timestamp

Pages at the top are what AI "knows best" about your site.

Crawler breakdown

See the distribution across crawlers:

• Pie chart showing visit share
• Table with detailed counts per crawler
• Trend indicators (↑ increasing, ↓ decreasing)

Training vs. real-time crawlers

Understanding the difference is crucial:

Training crawlers

GPTBot, ClaudeBot, Google-Extended collect content to train future model versions. What they crawl today may influence AI responses in 6-18 months. Note: Google-Agent is a separate, newer user-agent representing AI agents that take actions on your site (filling forms, navigating) - it's distinct from training crawlers.

• Visits don't mean immediate citations
• But no visits means you won't be in future training data
• Focus on making your best content accessible

Real-time crawlers

ChatGPT-User, PerplexityBot, Claude-User fetch pages in real-time when users ask questions. This is live retrieval.

• Visits often correlate with active citations
• Fresh content matters - they get your latest version
• Fast page loads improve retrieval success

Optimizing for crawlers

If you're not seeing crawler visits, or they're missing key pages:

Make content accessible

• Ensure pages aren't blocked by robots.txt
• Remove JavaScript-only content barriers
• Provide clean HTML with semantic structure

Don't block AI crawlers

Check your robots.txt for these common blocks:

# DON'T do this if you want AI visibility
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Consider AI Pages

AI Pages automatically optimizes your pages for AI crawlers - serving clean, structured content that AI can parse efficiently.

Requirements

To see AI crawler data, you need:

Without AI Pages, we can't see when crawlers visit your site.

Heatmap view

Toggle to the heatmap to see crawl patterns by hour and day of week:

• Rows = Days of the week
• Columns = Hours (in UTC)
• Color intensity = Number of visits

This reveals when AI platforms are most active crawling your site. Useful for:

• Understanding crawl schedules
• Timing content updates before major crawl windows
• Identifying unusual patterns

Data freshness

Click Refresh to clear cache and fetch latest data.

Exporting crawler data

Export your crawler visit history:

1. 1Select date range
2. 2Click Export → CSV
3. 3Download includes: timestamp, crawler, page URL, status

Useful for:

• Sharing with technical teams
• Correlation with content updates
• Audit and compliance

FAQ

Q: Why don't I see any crawler visits?

Most likely AI Pages isn't set up. Crawler detection requires the AI Pages edge proxy to be active.

Q: Is crawler activity the same as visibility?

Not directly. Crawling means AI is collecting data. Visibility means AI is recommending you. High crawl activity without visibility means AI knows about you but doesn't cite you yet.

Q: Can I block certain crawlers?

Yes, both via robots.txt and via AI Pages settings. But blocking reduces AI visibility.

Q: How far back does data go?

We store 90 days of crawler visit history.

Next steps