Is this really GPTBot?

Check its address, not its user-agent. OpenAI publishes the IP ranges GPTBot uses at openai.com/gptbot.json. A request outside those ranges is not GPTBot, whatever its user-agent says. The verifier fetches that list and checks the address for you.

Run a reverse DNS lookup on the address. A genuine Googlebot hostname ends in googlebot.com or google.com. Then look up that hostname and confirm it points back to the same address. Google also publishes its crawler IP ranges, and the verifier checks both.

Anthropic publishes one list of addresses for all its crawlers: ClaudeBot, Claude-SearchBot and Claude-User. The address proves the request came from Anthropic, and the user-agent says which bot it was.

Those are private addresses inside your own network, usually a load balancer or proxy. The crawler’s real address is in the X-Forwarded-For header or your CDN’s client IP field.

ChatGPT Agent, ExaSearchBot and ApifyBot sign each request with HTTP Message Signatures (RFC 9421). Their address proves nothing, so check the Signature and Signature-Agent headers against the operator’s published keys. Many CDNs, Cloudflare included, do this for you.

Yes: a request that claims to be a crawler but fails the check is not that crawler, so a robots.txt rule won’t stop it. Block or rate-limit it by address. For genuine crawlers, use robots.txt instead. Anthropic notes that blocking its addresses can stop its crawler reading your robots.txt at all.

Each list is fetched from the operator and refreshed every six hours, and the result shows the date the operator gives. Addresses you check are not stored.