Bytespider
Bytespider is ByteDance’s crawler used for training ByteDance’s AI models (TikTok’s owner).
What it means for your site
It collects pages to train models. Blocking it doesn’t remove your site from current answers, but it may reduce what future models know about your brand.
Should you allow or block it?
It’s your call. Blocking protects your content from model training; allowing helps models know your brand. It doesn’t directly affect your presence in search-based answers.
How to allow or block it
To allow it (in the robots.txt file at the root of your site):
User-agent: Bytespider Allow: /
To block it:
User-agent: Bytespider Disallow: /
Anyone can put a crawler’s name in their user agent. To confirm a visit is genuine, check the IP address against the list the company publishes. Blocks at the CDN or firewall don’t show up in robots.txt.
How to know if it visits your site
The robots.txt checker shows whether this crawler is allowed. With a Neural Ranks account, the AI crawlers page in your dashboard shows how often each crawler visits your site and which pages it reads.
See where your brand stands in AI search.
Enter your site, confirm the suggested topics and see your first real analysis across the 5 AI engines. Free, no card.