Re:Scrape like the big boys
by convolutionart on Friday November 05, 2021 @10:01AM
(#29119349)
This is nonsense. It's always easier to destroy than to build/mantain. If you got any real advice, by all means...
If an amateur can do damage to you, then I have some bad news for you...
I believe the point wasn't surprise that damage occurred at all, but frustration that damage can occur just out laziness/ignorance rather than malice.
Re:Scrape like the big boys
by scarygliders on Friday November 05, 2021 @10:36AM
(#29119874)
Indeed, that was precisely their point, and "bad news for you" is disingenuous as there are many techniques used by incompetent, or just downright unethical and greedy scraper companies which, no matter how robust the target is, can still give it a major headache.
I've witnessed a site being basically DOS'ed due to particularly greedy and aggressive mass scraping attempts.
Precisely this, thank you.
To be clear, they did “damage” was to our bottom line. Most sites don’t capacity plan for random cliff walls of 2-10x traffic (clearly we should!). We’re scalable enough to handle that traffic after a period, but a) it caused intermittent periods of low availability (costing us money because we didn’t generate income the way we normally do) and b) cost us money from scaling all our services up.
It’s just selfish. If you’re going to take the product of other people’s work in a manner they don’t consent to, at least do it in a way that doesn’t cost them twice over.
If an amateur can do damage to you, then I have some bad news for you...