Crawlers

AI crawler monitoring records web requests that match known AI crawler or fetcher signatures. Trakkr keeps the page, time, method, response result, purpose class, collection source and available identity evidence together.

That request is the first observation in a wider measurement funnel:

1. 1.A matching crawler request reaches your CDN, WAF, edge or origin.
2. 2.Trakkr normalizes the requested page and records the response evidence.
3. 3.Prompt tracking separately records whether an AI answer cited that page.
4. 4.Connected analytics separately records attributable AI referral sessions.
5. 5.Trakkr compares the observations by normalized page and time window, then suggests the next check.

A crawl does not prove indexing, model use, ranking, citation or a later human visit. A citation after a crawl is an association worth investigating, not proof that the crawl caused the citation.

Separate crawler purpose before changing access

The same provider can publish several tokens with different jobs. Do not apply one bot's purpose or robots policy to the whole provider.

OpenAI documents GPTBot, OAI-SearchBot and ChatGPT-User separately. Blocking GPTBot does not automatically block OAI-SearchBot. Anthropic similarly documents ClaudeBot, Claude-SearchBot and Claude-User as separate tokens. Perplexity documents PerplexityBot for search and Perplexity-User for user actions.

Google-Extended is not a crawler. Google describes it as a robots.txt control token for certain Gemini and Vertex AI uses. It has no separate HTTP user-agent, and Google says it does not affect Google Search inclusion or ranking.

Supported signatures

Trakkr's detector is pattern-based and most-specific-first. Active patterns include, among others:

Some strings in the long tail are reported rather than confirmed in current operator documentation. Trakkr's AI Crawler Observatory labels official, reported and observed evidence separately.

User-agent is not identity proof

Any client can copy a user-agent. Treat a name match as a classification clue. When it matters, compare the source IP with the operator's current published IP ranges, use reverse DNS or provider signatures where documented, and retain the verification method with the event.

Network verification is stronger evidence about the sender. It still does not show what happened after the response.

What Trakkr records

The exact fields depend on the collection source. A crawler event can include:

• normalized URL and pathname
• crawler label and purpose class
• event type and collection method
• request time
• HTTP method and response status
• referrer when supplied
• user-agent signal
• duration where the source exposes it
• bot or source type
• connection and brand scope

Trakkr can then build a page-level comparison with:

• search, user-fetch and training request counts
• bot breakdown and last recorded request
• response-status buckets
• observed citations and cited answer samples
• AI referral clicks or sessions when analytics is connected
• optional Google Search Console clicks, impressions, queries and position
• robots.txt, llms.txt and page-access findings
• a machine-generated review status and next action

These fields come from different evidence sources. They are joined on normalized page URLs and aligned windows. A missing value means no matching observation in the selected data, not proof that the event never happened.

Install methods

Choose the earliest safe request layer that exposes the evidence you need. The install guide routes you to the right setup.

You can connect more than one source, but overlapping feeds can count the same real request more than once when the sources do not share an event identity. Prefer one authoritative source per request path unless you are deliberately comparing layers.

Verify the setup in three levels

Do not treat one green test as proof of the whole path.

1. 1.Synthetic dashboard test: a known test event confirms that Trakkr can store, query and display a crawler row. The event is labelled as synthetic. It does not prove the provider source is installed.
2. 2.Source connection check: where supported, a provider-specific check confirms token scope, project or zone configuration, Log Drain state or delivery health.
3. 3.Real delivery check: a real matching request recorded through the installed source is the strongest proof that the live ingest path works.

If the synthetic row appears but real activity does not, inspect the source connection, robots policy, WAF order, cache layer and time window. Quiet real crawler traffic is also normal; you cannot force an operator to recrawl on demand.

Product views

The main Crawler dashboard includes these views:

• Pages: page-level request totals, crawler classes, citations, referrals, response health, last activity and a shared page drawer.
• Map: coverage across the site's page structure, including important pages with no matching request in the selected window.
• Live: recent classified events with method, status, path, source, time and collection method.
• Actions: review work based on access gaps, errors and crawl-without-outcome patterns.
• Access: robots.txt, llms.txt, status and outcome findings by bot token or class.
• Sources: connection setup, source checks, delivery health and install guidance.

The page drawer connects crawler evidence with citations, referral traffic, linked work and Timeline. The Pages index adds sitemap, Search Console, publishing and other discovery sources.

Access checks and controls

Trakkr diagnoses access evidence. It does not claim to control or block a provider merely because it can show a finding.

• robots.txt expresses policy for documented tokens. It does not authenticate the sender.
• WAF and CDN rules can block before the origin or analytics layer sees a request.
• HTTP status shows what the observed layer returned. A 200 response does not prove content parsing or use.
• llms.txt is an advisory proposal. It is not access control, not a Google ranking factor and not a guarantee that an AI provider will use a page.
• IP and reverse-DNS checks can strengthen sender verification where the operator publishes a method.
• Caching and sampling can make two valid systems report different totals.

Cloudflare Crawl Control and Vercel firewall rules are provider control surfaces. Trakkr can help you inspect their outcomes and decide what to review. Apply enforcement in the provider, host, WAF or robots file, using the provider's current documentation.

How to investigate a page

1. 1.Confirm the normalized URL and selected time window.
2. 2.Check which request layer collected the event and whether that layer samples data.
3. 3.Separate training, search and user-fetch signatures.
4. 4.Review method, status, robots policy, WAF order and cache behavior.
5. 5.Verify identity beyond user-agent when the decision affects access or security.
6. 6.Compare separately observed citations and AI referrals for the same page and window.
7. 7.If a page is crawled but not cited, review prompt relevance, answer evidence, content fit and competitors. Do not assume more crawling is the fix.
8. 8.If a page is cited but has no matching crawl, check collection gaps, older windows, cached indexes, alternate URLs and canonicalization.

Limits

• Trakkr sees only installed sources and their retained fields.
• WAF order, caching, reverse proxies and serverless routing change which layer sees a request.
• Provider analytics can be sampled or aggregated.
• User-agents can be spoofed and provider labels can change.
• A request can be retried, cached, redirected or logged more than once.
• A user-requested fetch does not expose the user's conversation.
• Request timing does not prove indexing, model training, ranking, citation or referral causality.
• Google Search Console and crawler logs measure different systems and should not be expected to match.

Common questions

Should I block GPTBot to stop OpenAI training use?

GPTBot is OpenAI's documented training crawler. OAI-SearchBot is the separate search crawler, and ChatGPT-User is the separate user-triggered fetcher. Set each policy deliberately. OpenAI says the GPTBot and OAI-SearchBot settings are independent, although it may reuse a crawl for both when both are allowed.

Does OAI-SearchBot indexing mean ChatGPT will cite the page?

No. Access and indexing can support eligibility. They do not guarantee ranking, retrieval or citation. Measure citations separately.

Does ChatGPT-User prove that my page was cited in a conversation?

No. It records a user-triggered fetch signature. The request log normally does not contain the question, answer or citation decision.

Why do Cloudflare and my origin log show different totals?

They observe different request layers. WAF order, cache hits, sampling, retries, time zones and classification rules can all change totals. Compare the same window and keep the collection method visible.

Is the synthetic test a real GPTBot visit?

No. It is a labelled verification row used to test Trakkr's storage and display path. Use the source check and a real delivered request to verify the installation.

Are crawler requests the same as AI visibility?

No. Requests, citations and referral visits are different observations. Trakkr connects them by page so you can find gaps and next checks without hiding that distinction.

Going further