It's not always about individual bad actors. You can have lots of small players causing problems. I wonder how many python developers there are right now trying to make their own offline copy of stackoverflow.com.
Wikipedia has a great defence against this. They ask you not to scrape, and at the same time, provide torrents of the data (https://meta.wikimedia.org/wiki/Data_dump_torrents)
There was a hiccup around June but that seems resolved now: https://meta.stackexchange.com/questions/389922/june-2023-da...
What is your opinion on spam email?
Some people don't publicly publish their email address, they instead selectively give it out only to those they want to get email from, but their address gets leaked/sold and abused. Ultimately people who do publicly publish a contact address (email or even a physical mailing address) are basically on the hook for deciding what to do with whatever people send them.
The spam situation got out of hand pretty fast though. The only thing that kept email spam from reaching the level of a DoS attack were blacklists and server-side filtering, and even with those things (plus client-side filters) spam is still a huge problem today. Spam is just a much bigger problem than web scraping. Even the junkmail the mailman delivers to my door has an environmental cost that's much worse than the "harm" of a web scraper's http GET requests.
We have many alternative ways to contact each other online that aren't as vulnerable to spam, but for all of its shortcomings email continues to be widely used because at the end of the day people think giving strangers the ability to reach out to them uninvited is valuable. Anyone can set up a whitelist and trash everything that comes into their mailbox unless it's from an approved sender, but almost nobody does because they want to be more reachable than that.
- it seem a lot of web developer / product manager either do not know or do not care about robots.txt
- some web application are so badly optimized, that some of them are not able to handle more than 1 hit per second at a sustained rate, which admittedly have worked fine so far. But crawlers are persistent, causing the normal crawling activity to cause denial of service for normal users.
that's the expected cost of publishing something to the public internet. People are going to access it. No one has a right to complain when people access something that was put there for the public to see. Scrappers can be dicks about it too, they can get lazy and endlessly hammer away at some server or repeatedly pull down the same content because they messed up, but we don't need need litigation for that. If something raises to the level of DoS that's already covered under existing laws.
> it's completely in keeping with what amoral corporations locked in constant battle would be expected to do: maximize benefits to themselves while minimizing benefits to others.
Maybe we need to rethink giving some of these corporations the privilege of corporate personhood if they are just going to make things worse for everyone else while only enriching themselves. We don't need to allow parasites and pillagers to take whatever they want at our expense.