Re:ByteDance’s Bytespider is scraping at much higher rates t...
by jsheard on Monday October 07, 2024 @08:02AM
(#41765145)
> They are also careful to shard per domain and only crawl each domain at a small rate (~1 qps) as to not ddos the site.
Maybe that's their intent, but this was only a month ago: https://www.gamedeveloper.com/business/-this-was-essentially...
> "The homepage was being reloaded 200 times a second, as the [OpenAI] bot was apparently struggling to find its way around the site and getting stuck in a continuous loop," added Coates. "This was essentially a two-week long DDoS attack in the form of a data heist."
Re:ByteDance’s Bytespider is scraping at much higher rates t...
by jefozabuss on Monday October 07, 2024 @08:41AM
(#41765427)
Maybe someone went against the rule of deploying on a Friday, ouch.
They have dedicated user agents for search crawling, when a user directly asks about a site and for training data.