Re:Congrats! Web scraping is legal! (US precedent)
by hannasanarion on Wednesday January 29, 2020 @12:47PM
(#22183502)
So you throttle your users. We have http status codes for "too many requests" and all scraper software comes with a delay setting by default. Everybody who does scraping is supposed to know that its rude to blast a thousand requests per second.
[ Parent
| on HN ]
Re:Congrats! Web scraping is legal! (US precedent)
by munk-a on Wednesday January 29, 2020 @01:07PM
(#22183791)
This ruling has left open a big question of how much you need to spend to support scrapers and where the line between scraping and a DoS attack lies - and that's going to be a weird line. If my site is producing a big report off of data that changes quarterly then re-downloading that report every 20 minutes is possibly excessive and might wander into the realm of an attack - while if we looked at the same frequency with twitter it seems a lot more reasonable - maybe even a bit on the slow side.