I got a robots.txt or crawler-blocking alert from Trakkr that I think is wrong. How do I check it myself?
Updated June 30, 2026 · Reviewed by [email protected]
Start by checking your live robots.txt directly (not a cached version) for any Disallow rule that covers GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, or Google-Extended. A common gotcha is a CDN or edge cache serving an old robots.txt even after you've updated the origin file, so it's worth checking through a fresh, uncached request.
If robots.txt looks clean, the alert is more likely coming from a 403 or challenge page on the actual page fetch rather than a robots.txt rule. Those are two separate checks, so it helps to know which one triggered the alert before assuming it's wrong.
Related questions
- Crawler tracking shows visits from some AI bots (like Meta or Mistral) but not others (like ChatGPT or Perplexity), even though I know those engines have visited. Is tracking broken?
- Why does Trakkr say my site is blocking AI crawlers when I can see GPTBot traffic getting through in my own logs?
- Which AI bot user agents should I allow in my firewall or WAF so Trakkr can track them?
- Can I install crawler tracking on a site that isn't Cloudflare, Vercel, Netlify, or WordPress (like Wix, Framer, or a custom PHP site)?
- Connecting my Vercel-hosted site to crawler tracking keeps failing. What's going on?