Crawlers
Crawlers records web requests that match a known AI crawler signature and keeps the page, time, method, response status, purpose class, collection source and user-agent together. It is the only place in Trakkr measured from your own server-side records rather than by asking a model a question.
Open it to answer three things: can AI reach this page at all, which pages are the bots actually reading, and are any of them being turned away. It needs a paid plan and at least one connected source. If you have not connected one yet, start with the install guide.
A request is evidence that a fetch happened. It is not evidence of indexing, training, ranking, citation or a later human visit. A citation after a crawl is worth investigating, not proof that the crawl caused it.
The five purpose classes
The same provider publishes several tokens with different jobs, so never apply one bot's policy to a whole company. Trakkr sorts every request into one of five classes, and the dashboard groups them as live demand (Conversations, Agents) versus background collection (Indexing, Training).
| Class | What the operator says it does | Examples |
|---|---|---|
| Training | Content may be used to improve models | GPTBot, ClaudeBot, CCBot, Amazonbot, Bytespider |
| Indexing | Builds or refreshes a retrieval index used by AI search | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot |
| Conversations | Fetches a page during a user's session | ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User |
| Agents | Acts for a tool rather than a person reading a chat | Google-Agent, Claude-Code |
| Other | Documented, but none of the above | OAI-AdsBot |
OpenAI documents GPTBot, OAI-SearchBot and ChatGPT-User separately, and blocking one does not block the others. Anthropic does the same with ClaudeBot, Claude-SearchBot and Claude-User. Perplexity separates PerplexityBot from Perplexity-User.
Google-Extended and Applebot-Extended are not crawlers. They are robots.txt control tokens with no HTTP user-agent of their own, so Trakkr's detector cannot emit them and they never appear as traffic.
What Trakkr detects
The detector is a pattern list matched most-specific-first, so ChatGPT-User resolves before the generic ChatGPT fallback. It currently covers more than thirty signatures, including:
| Provider | Signatures |
|---|---|
| OpenAI | GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot |
| Anthropic | ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, Claude-Code, anthropic-ai |
| Perplexity | PerplexityBot, Perplexity-User |
| Apple, Meta, Mistral, Google | Applebot, Meta-ExternalFetcher, MistralAI-User, Google-Agent |
| Long tail | Amazonbot, Bytespider, CCBot, GrokBot, DeepSeekBot, cohere-ai, AI2Bot, Diffbot, Timpibot, YouBot, ImagesiftBot and others |
Some strings in the long tail are reported rather than confirmed in current operator documentation. The AI Crawler Observatory labels official, observed and reported evidence separately, and lists the verification method each operator publishes.
Two adjustments run when Trakkr reads the data back, so they apply to old rows too. A request carrying claude-code in its user-agent is reclassified as Claude-Code, an agent rather than a conversation. And a specific malformed variant of the ChatGPT-User string is relabelled Unverified-ChatGPT-User, because over thirty days it requested security-probe paths eight times more often and static assets eighty-seven times more often than the genuine string.
Meta AI and You.com requests are recognised but hidden from every dashboard view.
A user-agent is not proof of identity
Any client can copy a user-agent string, and Trakkr's pipeline does not verify sender IPs: the Cloudflare analytics dataset is aggregated, so no client IP arrives with the event. Treat a name match as a classification label.
Two things do make the picture more honest. Trakkr excludes known security-probe paths from crawler counts, since spoofed scanners request them while forging crawler names. And where an operator publishes IP ranges, reverse DNS or signed requests, you can verify the sender yourself before making an access or security decision. Network verification tells you who sent the request; it still says nothing about what happened after the response.
The five views
The Crawlers page has five tabs, over a window of 24 hours, 7, 30 or 90 days.
- Pages. Page-level request totals, crawler classes, citations, referrals, response health and last activity. Opens the shared page drawer.
- Map. Coverage across the site's structure, including important pages with no matching request in the window.
- Live. Recent classified events with method, status, path, source, time and collection method.
- Actions. Review work generated from access gaps, errors and crawl-without-outcome patterns.
- Access. robots.txt, llms.txt, status and outcome findings, by bot token or class.
Sources lives in the header menu and opens a separate page for connections, health and install guidance.
The page drawer is where the comparison happens. It puts crawler evidence next to observed citations, referral traffic, linked work and the page timeline. With a Search Console property connected, it also shows that page's Google position, clicks, impressions and top query for the same window. The fields come from different systems joined on the normalized page URL, so a missing value means no matching observation, not proof the event never happened.
Access findings
The Access tab diagnoses evidence. It does not block anything; enforcement belongs in your robots file, WAF or CDN.
| Finding | What triggers it |
|---|---|
| High-value blocked | A search or conversation bot is blocked and cited pages would be affected |
| Robots block | The bot is disallowed in robots.txt and Trakkr still saw denied requests from it |
| Access mismatch | robots.txt allows the bot, but over half its requests failed (denied, missing, or server errors) across at least ten attempts |
| Traffic dropped | Requests fell more than 70% against the previous period with no robots rule to explain it, backed by either 403/429 responses now or sustained traffic before |
| llms.txt | No llms.txt found |
| Forged identity | Requests carrying a crawler name that the evidence does not support |
A few things worth holding onto while you read them. robots.txt expresses policy for documented tokens but does not authenticate the sender. A 200 response proves the bytes were served, not that they were parsed or used. llms.txt is an advisory proposal, not access control and not a ranking factor. And caching or sampling can make two correct systems report different totals.
Investigating a page
- Confirm the normalized URL and the window, and check whether the collecting layer samples.
- Separate the classes. Training-only traffic is a different story from Indexing or Conversations.
- Review method, status, robots policy, WAF order and cache behavior.
- Verify identity beyond the user-agent when the decision affects access or security.
- Compare the separately observed citations and referrals for the same page and window.
If a page is crawled but never cited, more crawling is not the fix. Look at prompt relevance, the evidence on the page, content fit and what competitors offer instead. If a page is cited but has no matching crawl, check for collection gaps, older windows, cached indexes, alternate URLs and canonicalization.
Limits
- Trakkr sees only the sources you install and the fields they retain.
- WAF order, caching, reverse proxies and serverless routing all change which layer sees a request.
- Provider analytics can be sampled or aggregated, and Cloudflare's lookback depends on your plan.
- User-agents can be forged, and provider labels change.
- A request can be retried, cached, redirected or logged more than once.
- A conversation fetch never exposes the question that caused it.
- Search Console and crawler records measure different systems and should not be expected to match.
Common questions
Should I block GPTBot to stop OpenAI training use?
GPTBot is OpenAI's documented training crawler, and it is separate from OAI-SearchBot for search and ChatGPT-User for user-triggered fetches. Set each policy deliberately. OpenAI says the GPTBot and OAI-SearchBot settings are independent, though it may reuse a crawl for both when both are allowed.
Does an OAI-SearchBot crawl mean ChatGPT will cite the page?
No. Access and indexing support eligibility; they do not guarantee retrieval or citation. Measure citations separately and compare the two in the page drawer.
Does ChatGPT-User prove my page appeared in a conversation?
No. It records a fetch that carried a user-triggered signature. The request never contains the question, the answer or the citation decision.
Why do Cloudflare and my origin log show different totals?
They observe different layers. WAF order, cache hits, sampling, retries, time zones and classification rules all shift the count. Compare the same window and keep the collection method visible before concluding either is wrong.
Why do I never see Meta AI or You.com requests?
They are detected but hidden from every dashboard view, along with the Google-Extended and Applebot-Extended control tokens, which are opt-out signals rather than fetches.
Are crawler requests the same thing as AI visibility?
No. Requests, citations and referral visits are three different observations. Trakkr connects them by page so you can find the gaps, without pretending one implies another.