Install crawler tracking

Crawler tracking reads server-side request records, so it has to see the request before a browser would. Pick the deepest layer you can connect: a CDN or hosting integration sees cache hits and requests that never reach your app, while an origin integration sees only what reaches the origin.

Start in the product. Open Traffic → Crawlers, choose Sources from the header menu, then Add source. The picker there matches the current catalog and generates the endpoint, token or template your platform needs. This page is the reference for what each platform's setup involves and what it can and cannot see.

Cloudflare {#cloudflare}

Works when the site's DNS record is proxied through Cloudflare (the orange cloud). Trakkr reads Cloudflare's httpRequestsAdaptiveGroups analytics with a scoped read-only API token, so there is no DNS change, no firewall change and no code to deploy. Every plan including Free exposes enough analytics.

Create a custom token at Cloudflare API Tokens with exactly three read permissions. The Trakkr modal links a pre-filled template that selects them for you.

Scope Zone Resources to all zones or just the domain you want, then create the token and copy it. Cloudflare shows it once. Back in Trakkr, paste it, choose Verify token, pick the zone and Connect zone.

Trakkr pulls from Cloudflare roughly every hour. How far back it can look depends on your Cloudflare plan: one day on Free, seven on Pro, thirty on Business, ninety on Enterprise. The analytics layer can also aggregate or sample, so treat long-tail daily counts as estimates.

If you need per-request delivery, the connection card offers an optional Cloudflare Worker template. It captures matching requests that reach the Worker, but it cannot recover a request a higher-priority WAF rule stopped first.

If the token verifies but no real activity appears, check that the record is still proxied, then look at Bot Fight Mode and custom WAF rules in Trakkr behind a WAF.

Vercel {#vercel}

Vercel needs a Pro or Enterprise plan, because Log Drains are not available on Hobby. If you are on Hobby, use the Next.js method instead; middleware works on any plan.

Choose Vercel in the picker and Continue to Vercel. Vercel opens its marketplace install screen for the Trakkr integration; the permissions are in the integration manifest, so there is nothing to toggle. Pick the team that owns the project, install, and Vercel returns you to Trakkr. Then pick one project and Trakkr creates the Log Drain for you.

The drain is scoped to production traffic on that project, meaning your main domain and its production aliases. Preview and branch deployments are excluded. One Vercel project connects per brand; picking another pauses the first.

Every request generates a log line to Trakkr's ingest endpoint. Trakkr inspects the user-agent, keeps matching AI crawler hits and discards everything else on arrival, so human visitor data is never stored.

The connection card has a Check connection button. It re-reads the drain from Vercel and tells you whether it exists, is enabled, is scoped to the right project and points at the right endpoint, and whether a delivery has arrived yet. Disconnecting in Trakkr deletes the drain from Vercel in the same step.

Netlify {#netlify}

Netlify uses OAuth to pick the site, then one Edge Function file that you deploy. Trakkr can list your sites; it does not read site content, deploy code or change DNS, and it does not push anything into your repo.

Choose Netlify, Continue to Netlify, approve access for the right team, then pick the site. Trakkr creates a pending connection and shows the template plus its credentials.

Create this file in the connected site's repository and paste in the template from the connection card:

netlify/edge-functions/trakkr-crawler.ts

Netlify picks up that path automatically, with no netlify.toml change. Then add three values in Site configuration → Environment variables:

TRAKKR_CONNECTION_ID=<your connection id>
TRAKKR_WEBHOOK_SECRET=<your webhook secret>
TRAKKR_INGEST_URL=https://api.trakkr.ai/crawler-connect/ingest/netlify

Commit and push, or use Deploys → Trigger deploy → Clear cache and deploy site. A new deploy is required before the function can receive traffic. The function lets every request through first and only then fires a background POST for matching crawlers, so it adds no perceptible latency and never changes your HTML or headers.

The connection stays pending until the first authenticated POST from the function arrives. If the function is live but nothing lands, search the Netlify deploy logs for TRAKKR_ and fix any undefined environment variables, then redeploy.

To remove it: disconnect in Trakkr, delete the file, deploy again, and optionally revoke Trakkr under Netlify's authorized apps.

WordPress {#wordpress}

A small plugin runs at the WordPress origin on every request. It adds no front-end script, so there is no page-weight or Core Web Vitals cost. You need WordPress 6.3 or later on PHP 7.4 or later, served over HTTPS, and an admin account to do the install.

1. Choose WordPress in the picker and download the plugin, or use the permanent link https://api.trakkr.ai/crawler-connect/wordpress/plugin.
2. In wp-admin, open Plugins → Add New → Upload Plugin, upload the ZIP and activate it. On WordPress VIP, add the unpacked folder through the application's Git repository, deploy, then activate.
3. Open Users → Add New and create a user such as trakkr-sync with the Trakkr Crawler Sync role. The plugin adds that role on activation.
4. Open that user's profile, scroll to Application Passwords, add one named Trakkr and copy it. WordPress shows it once.
5. In Trakkr, leave the connection type on Crawler only, enter the HTTPS site URL, the username and the Application Password, then Enable tracking. Pick Existing publishing instead if the site is already connected under Sites and you want to reuse that account.

The crawler-only role holds three capabilities: read, retrieve crawler rows, and mark retrieved rows as synced. It cannot publish content, edit posts, upload media, change site options or write files. An administrator also works, because the plugin grants administrators the same two crawler capabilities, but it is more access than tracking needs.

Trakkr pulls new rows over the plugin's /wp-json/trakkr/v1/ REST routes about every four hours. Inside WordPress, synced rows are deleted after 30 days and unsynced rows after 90, so the table cannot grow without bound.

The plugin sees only requests that reach WordPress. A CDN or WAF can serve or block a request first. If your security tool locks down /wp-json/, allow authenticated access to /wp-json/trakkr/*. The WAF guide has the per-platform steps.

Next.js self-hosted {#nextjs}

For a Next.js app you host yourself. Trakkr generates an ingest endpoint, a bearer token and a proxy or middleware template.

Name the connection, copy the endpoint, token and template, and send the built-in sample event to check authentication before you deploy anything. Then add the template to your proxy or middleware path and deploy. Keep the token in a server-only secret, set as TRAKKR_BEARER_TOKEN.

The template forwards matching hits in the background and does not delay the response. If the app sits behind a CDN, connect the CDN instead so cache hits are not missed.

Node and Express {#node}

Express middleware that reports a crawler request after the response finishes. Works with Express, Fastify, Koa, NestJS or any Node HTTP server.

Create the connection, copy the endpoint, bearer token and middleware, send the sample event, then add the middleware before your routes and deploy with the token in a server secret.

Behind a reverse proxy, set Express trust proxy correctly so the reported IP comes from the real client header. The IP is used for analytics only, never as proof that a bot is genuine.

Nginx and OpenResty {#nginx}

An asynchronous OpenResty log hook, using log_by_lua_block and lua-resty-http.

Copy the generated endpoint, bearer token and configuration, send the sample event, add the hook to the request log phase, validate the config and reload Nginx.

Set real_ip_header and your trusted proxy ranges before relying on the client IP when Cloudflare, Sucuri or another proxy sits in front. A plain Nginx origin cannot see requests served or blocked at the CDN.

AWS CloudFront {#cloudfront}

A Lambda@Edge function on Viewer Request, so it runs before the cache decision.

Copy the generated template, send the sample event, then create the function in us-east-1, publish a numbered version and attach it to the distribution's Viewer Request event. Wait for the association to deploy.

Lambda@Edge does not support environment variables, so the endpoint and bearer token live in the function source. Restrict access to the function, and recreate the connection if the token is exposed.

Akamai {#akamai}

For DataStream 2 or another Akamai HTTPS forwarder. Akamai's connector does not support bearer tokens, so Trakkr issues Basic Authentication credentials instead: the username is the connection id and the password is the webhook secret.

Create the connection, copy the template, send the sample event, then configure DataStream 2 to POST matching request batches to the endpoint with those credentials. Akamai forwards in batches, typically within about ten minutes.

Fastly {#fastly}

Real-Time Log Streaming over HTTPS with a bearer token in the Authorization header.

Send the sample event first, add the HTTPS logging endpoint in Fastly, then activate the service version. Fastly checks domain control before it will stream; Trakkr answers that check at https://api.trakkr.ai/.well-known/fastly/logging/challenge, so no action is needed on your side. Filter at the edge where you can, so only crawler rows are forwarded.

Other or custom edge {#custom}

The generic webhook covers any other CDN, reverse proxy or log forwarder. The setup screen gives you a POST endpoint, a bearer token, a sample payload and a curl example.

Events are a small JSON list carrying the time, URL, user agent, IP, status code and country where available. A JSON array, a single object or NDJSON all work, and the body may be gzip compressed. Trakkr accepts several common field names for each value, so a provider's native log shape usually needs little reshaping.

There is also a dry-run endpoint at https://api.trakkr.ai/crawler-connect/ingest/manual/validate. It takes the same payload and the same credentials, tells you how many entries parsed and how many were recognised as bots, and writes nothing. Use it while you are shaping the forwarder.

Hosted CMS platforms {#cms}

Some hosted sites can use Cloudflare in front of the CMS to collect crawler analytics. Check your host's requirements before changing DNS. Shopify, Wix and Ghost(Pro) do not support this route; use the alternatives below.

Cloudflare tracking works on Free. A host may require a paid plan or a specific reverse proxy setup. Cloudflare also provides SSL, caching, DDoS protection and AI bot controls.

For a supported setup:

1. Confirm the host's requirements below and prepare its approved records or proxy.
2. Recreate every existing DNS record in Cloudflare, including mail and other subdomains, before changing nameservers. Nameserver changes can take up to 48 hours.
3. Set the supported website records to Proxied. Use Full (strict) under SSL/TLS unless your host's guide requires a different configuration.
4. In Cloudflare's AI bot settings, choose Allow (do not block) for the bots you want to reach your site. Review managed robots.txt rules too. Tracking does not change access rules. See Cloudflare's bot settings guide.
5. Test the public site, then choose your platform in Trakkr and finish the Cloudflare token and zone steps.

Already using Cloudflare? Confirm the domain is proxied in your own Cloudflare account before skipping DNS setup. A host can use Cloudflare itself, so a detected Cloudflare response alone does not prove your account can read the traffic.

Webflow {#webflow}

Webflow's Orange-to-Orange setup needs a paid Webflow Site plan, but works with Cloudflare Free.

1. Open Site settings → Publishing → Production. If records still point to proxy-ssl.webflow.com or two legacy A records, migrate to Webflow's current DNS first.
2. Create proxied CNAME records for @, www, and each extra Webflow-served subdomain, all pointing to cdn.webflow.com. Do not proxy A records; this causes a 525 handshake error.
3. Keep SSL/TLS on Full (strict) and finish the shared setup above.

Webflow's Publishing panel keeps showing Update needed after proxying. That is expected. If traffic is missing, check the website records are still Proxied in your Cloudflare account.

Shopify {#shopify}

Shopify does not support putting a Cloudflare proxy in front of a hosted store. Do not change DNS for this integration.

Use Visitors to measure people arriving from AI answers and Citations to see where AI references your store. For a headless storefront, connect its actual hosting provider, such as Vercel or Netlify.

HubSpot {#hubspot}

Confirm your account's supported route with HubSpot before changing DNS. Its reverse proxy setup guide specifies Cloudflare Enterprise for O2O. Follow its domain, origin and proxy instructions; a nameserver change alone is not enough.

Once HubSpot confirms the setup and the public site works through your Cloudflare zone, connect that zone in Trakkr.

Squarespace {#squarespace}

Use Squarespace's external-domain connection and preserve the A and CNAME records it provides. Keep the verify.squarespace.com verification CNAME on DNS only. Squarespace warns that proxying may interfere with the connection, so test the site before connecting Trakkr. Preserve mail records before changing nameservers.

Wix {#wix}

Wix does not support proxied DNS records. Keep its records on DNS only. Use Visitors for AI referral traffic and Citations for references to your site.

Framer {#framer}

Confirm reverse proxy eligibility with Framer first. Its Cloudflare guide specifies Enterprise and uses a Worker, not a proxied CNAME alone.

1. Publish to your framer.website subdomain and set your canonical URL to the public domain.
2. Follow the guide to create a proxied dummy A record pointing to 192.0.2.1 for the hostname the Worker will serve.
3. Deploy Framer's forwarding Worker, assign its route to your public hostname, and turn off Rocket Loader. The Worker must be live before visitors use that hostname.
4. Test the site, review AI bot access, then connect the Cloudflare zone in Trakkr.

Ghost {#ghost}

Ghost(Pro) requires DNS-only records. A proxy can prevent certificate renewal. Use Visitors and Citations instead.

For self-hosted Ghost, connect the server using Nginx or Node. You do not need to move DNS to collect origin requests.

AI Pages {#ai-pages}

If AI Pages is enabled for the brand and already receiving crawler traffic, Trakkr can connect it as a source with no token, DNS change or code file. Choose AI Pages in the picker.

The option only appears when AI Pages is enabled and has recent traffic, and when it is not already connected. Trakkr pulls from it every thirty minutes. AI Pages does not store user agent, IP or country, so those fields stay empty on its rows. If the option is missing, connect the platform that actually receives your site's requests.

Confirm it is working {#verify}

There are three separate things to check, and they prove different things.

If test rows appear but real traffic stays empty past the source's expected delay, work through these in order:

1. The connection health message on the source, and the platform's own logs.
2. Whether the connected source actually sees the production hostname.
3. robots.txt rules for the missing bot, on the Access tab.
4. WAF, bot-management, rate-limit and security-plugin logs.
5. Whether two sources watch the same origin and should be reduced to one.

Quiet crawler traffic is also normal. You cannot make an operator recrawl on demand.

Common questions

Which method should I pick if more than one fits?

The one closest to the request. A CDN or hosting integration sees cache hits and requests blocked before your app runs; an origin integration does not. If you are behind Cloudflare, connect Cloudflare rather than the origin, even when the origin is easier.

Can I connect two sources for the same site?

You can, but avoid it unless you are deliberately comparing layers. The sources do not share an event identifier, so the same real request can be counted twice.

Why does the test ping succeed while real crawlers never appear?

Because the test ping writes rows directly into Trakkr's store. It exercises storage and display, not your connector, your network path or your firewall. Treat it as proof the dashboard works and nothing more.

Does any of this slow my site down?

No. Every method reports after the response has been sent or reads records the platform already keeps. Nothing runs in a visitor's browser, and nothing sits in the critical path of a page load.

What happens if I disconnect?

Data collection stops and previously collected events stay. For Vercel, Trakkr also deletes the Log Drain. For Netlify you should delete the edge function file and redeploy. For WordPress, deactivating and deleting the plugin drops its local table, its options and the Trakkr Crawler Sync role.