Bytespider
Also called: ByteDance crawler, ByteSpider, Bytespider bot
Bytespider is the web crawler run by ByteDance, the company behind TikTok. It gathers web content to train ByteDance's large language models (including its ChatGPT rival Doubao) and to feed other ByteDance AI products.
You can spot Bytespider in your logs by its user-agent, which carries a ByteDance contact address:
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)Unlike Googlebot or Bingbot, Bytespider does not send traffic back to your site. It reads your pages to build training corpora, so a visit is a pure resource cost with no click in return.
The robots.txt question
Bytespider earned a bad reputation in 2023 and 2024, when site owners and independent trackers reported it crawling paths that were explicitly disallowed. ByteDance has since said the crawler honors robots.txt, and agent directories now list it as expected to comply. Treat that as improving, not settled. Because a Disallow line is a request and not a wall, many operators block Bytespider at the server or CDN layer (Nginx, Apache, Cloudflare) so a 403 is returned before any HTML is served.
Blocking is a real decision, not a reflex. As of mid-2026 only a minority of top sites block it (one tracker put the figure near 17%). If ByteDance surfaces are a channel you care about, you may want the crawler in. If you are protecting proprietary content with no upside from ByteDance, you may not.
How it affects your traffic
Bytespider is one of the pipes feeding ByteDance's AI systems, so whether it can read your pages shapes whether your brand can surface inside Doubao and other ByteDance-powered answers. Block it wholesale and you may cut yourself out of that surface; leave it wide open and you hand over content with no traffic in return. Getting that trade-off right, crawler by crawler and page by page, is the AI crawler-access and visibility work in our AI SEO service.
Get AI SEO that moves the needle
We turn terms like this into ranked pages and qualified pipeline. Start with a free Initial SEO Strategy.