Bytespider
ByteDance’s crawler (the company behind TikTok and the Doubao assistant). Collects pages used to train and power its AI products.
What Bytespider is for
Collects pages to train future models. No direct traffic back to you.
High-volume and broad. Add a Bytespider directive to control it; compliance has been inconsistent historically, so verify in your logs.
Crawler guide
Evergreen identity, source, and robots.txt handling guidance.
Allow or block Bytespider
Partially honours robots.txt. Add one of these to your robots.txt.
User-agent: Bytespider Disallow: /
User-agent: Bytespider Allow: /
Bytespider has a documented history of ignoring robots.txt. ByteDance now publishes a Bytespider token, but confirm it is honoured against your own server logs.
Other AI crawlers
Every bot Trakkr tracks is a doorway.
AmazonbotOther · AmazonCommon questions
What is Bytespider?
Bytespider is an AI web crawler operated by ByteDance. ByteDance’s crawler (the company behind TikTok and the Doubao assistant). Collects pages used to train and power its AI products.
How do I block Bytespider?
Add a directive to your robots.txt: "User-agent: Bytespider" followed by "Disallow: /". Partially honours robots.txt. Bytespider has a documented history of ignoring robots.txt. ByteDance now publishes a Bytespider token, but confirm it is honoured against your own server logs.
Does blocking Bytespider hide me from AI?
Blocking Bytespider keeps your content out of training data but does not remove you from AI search answers - those are served by the search crawlers. High-volume and broad. Add a Bytespider directive to control it; compliance has been inconsistent historically, so verify in your logs.
Identity and robots.txt facts come from ByteDance's published bot documentation. Behaviour is measured from identified Bytespider requests in the server logs of the brands Trakkr tracks - 576K crawler visits across 84 sites, recounted as new data arrives.
Telemetry updated Feb 1, 2026.