News for hackers. Stuff that matters.
Re:Congrats! Web scraping is legal! (US precedent) by hannasanarion on Wednesday January 29, 2020 @12:47PM (#22183502)
So you throttle your users. We have http status codes for "too many requests" and all scraper software comes with a delay setting by default. Everybody who does scraping is supposed to know that its rude to blast a thousand requests per second.
Re:Congrats! Web scraping is legal! (US precedent) by munk-a on Wednesday January 29, 2020 @01:07PM (#22183791)
This ruling has left open a big question of how much you need to spend to support scrapers and where the line between scraping and a DoS attack lies - and that's going to be a weird line. If my site is producing a big report off of data that changes quarterly then re-downloading that report every 20 minutes is possibly excessive and might wander into the realm of an attack - while if we looked at the same frequency with twitter it seems a lot more reasonable - maybe even a bit on the slow side.