Trakkr behind a WAF

A WAF or security plugin is a common reason crawler data looks lower than expected. To many rule engines, AI crawlers look like scrapers, so the same defences can block them.

This page covers the three things that matter when Trakkr sits behind one: which install method to pick, how the real client IP makes it through proxy layers, and how to allowlist AI bots per platform.

Pick your install method

Pick the deepest layer Trakkr can read directly. CDN-side integrations see traffic that origin-side installs can't, so they're the better choice when available.

What each install method sees

When your site sits behind a CDN or caching WAF, requests served from cache never reach your origin. Origin-side integrations such as the WordPress plugin, Express, and Nginx do not record them. The same applies to requests the WAF blocks before they hit your stack.

The size of the origin-side gap depends on your cache keys, WAF rules and routing. Compare a known time window at both layers before choosing one source as authoritative. No install method can promise full coverage when an upstream system samples, aggregates or drops a request first.

Real client IP detection

When a request flows through a proxy, the TCP connection your origin sees comes from the proxy, not the client. The real IP rides along in a request header, and there is no single standard.

The Trakkr WordPress plugin walks this list and picks the first valid IP:

CF-Connecting-IP → True-Client-IP → X-Sucuri-ClientIP → X-Real-IP → X-Forwarded-For → REMOTE_ADDR

So a WordPress site behind Sucuri correctly resolves the real bot IP from X-Sucuri-ClientIP, even though every TCP connection comes from a Sucuri edge server.

Other origin-side integrations ship with the common headers but may need a one-line tweak when you're behind a WAF whose header isn't in the default set:

Review WAF treatment safely

Synthetic verification does not pass through your provider source, so it cannot show whether a real crawler is blocked. If real traffic stays empty beyond the source's expected delay, use these labels to search WAF and request logs:

GPTBot
ChatGPT-User
OAI-SearchBot
ClaudeBot
Claude-User
Claude-SearchBot
PerplexityBot
Perplexity-User
Bytespider
CCBot
Amazonbot
MistralAI-User
Meta-ExternalFetcher
Google-Agent
Applebot

Sucuri Firewall

1. 1.Open Sucuri dashboard → your site → Security Events and search the labels above.
2. 2.Confirm the source with operator-published network evidence where available.
3. 3.If policy allows the verified crawler, create the narrowest exception Sucuri supports for that source and path. Do not allow a broad user-agent regex through sensitive routes.

Cloudflare

Multiple bot layers can challenge crawlers. Handle whichever you're using:

• Bot Fight Mode: review Security Events to see whether the mode challenged the request. Changing the global mode affects more than AI crawlers, so use a bounded diagnostic window only when your security policy permits it.
• Bot Management: prefer Cloudflare's verified-bot or bot-management fields over raw user-agent text. Coverage varies, so compare the classification with the operator's current verification data.
• Custom WAF rules: inspect rule order first. If you add a Skip or Allow exception, require stronger verification than a copied user-agent and exclude authentication, checkout, admin and other sensitive paths.

Wordfence (WordPress)

1. 1.In Wordfence → Firewall → Blocking → Advanced Blocking, check for any User-Agent blocks matching AI crawler strings.
2. 2.In Wordfence → Live Traffic, filter by user-agent (for example GPTBot or ClaudeBot). Rows tagged Blocked are your culprits.

Several major operators publish current IP-range files or other verification methods. Use those where available, and keep them refreshed. Where no operator method exists, treat the user-agent as an observation label rather than identity proof.

iThemes / Solid Security (WordPress)

1. 1.In Security → Settings → Network Brute Force Protection → API Settings → Banned Hosts and Banned User Agents, remove wildcard matches like *bot* or *scraper* that catch AI crawlers.
2. 2.Check the Firewall section for User Agent filters under System Tweaks.

AWS WAF

1. 1.AWS WAF & Shield → Web ACLs → your ACL.
2. 2.If you use a managed rule group (AWSManagedRulesCommonRuleSet, AWSManagedRulesBotControlRuleSet), use Count temporarily to diagnose the matching rule when your security policy allows it.
3. 3.Create an exception only after verifying the source. Do not add a higher-priority Allow rule based only on an AI bot user-agent regex.

Imperva, StackPath, generic WAFs

Most enterprise WAFs can report the rule and classification that handled a request:

1. 1.Search security events by the claimed user-agent and time window.
2. 2.Verify the source with operator evidence where available.
3. 3.If policy permits it, create a narrow exception tied to the strongest available identity signal, path and method. The provider may call this Allow, Bypass or Skip.

Find out who's blocking

If you're not sure which layer is blocking, curl your site as a bot from outside the WAF:

"text-[#5b5fc7]">curl "text-accent">-A "Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)" \
  "text-accent">-I https://your-site.com/

This probe only shows how the site treats a request carrying that user-agent from your current network. It does not impersonate or verify GPTBot. A 403, 503 or challenge page can identify a policy path to investigate; a 200 does not prove that the real operator can reach the page. Response headers can help locate the handling layer.

Troubleshooting

Three places to look when you suspect a WAF is interfering:

1. 1.Send verification creates labelled synthetic rows and confirms that Trakkr can store and display them. It does not pass through your Cloudflare, Vercel, Netlify, WordPress or origin source, so it cannot isolate a WAF problem.
2. 2.Access tab cross-references your robots.txt with actual bot visit data. Two findings here flag WAF problems:

- Traffic dropped means visits fell sharply with no robots.txt change. Usually a new Cloudflare bot mode, WAF rule, or rate limit is responsible. - Access mismatch means robots.txt allows the bot but most of its requests get denied at the origin.

1. 1.robots.txt check parses your robots.txt and flags a Disallow: / under User-agent: GPTBot and similar entries. Bots respect this even if your WAF allowlist is correct.

Going further