Source: [email protected]
A site I worked on changed our captcha, and in the 24 hours after, we had a 90% reduction in new accounts. We had a full API including webhooks, the only private data was on the account page, there was no reason to scrape, but even if you did, there was no reason to be logged in... these accounts were used to harass, intimidate and scam legitimate users.
Businesses use data from the user. The Business does additional crunching on that data to derive new interesting data for the user. Who owns the data? The user or the app?
At the very least the user partially owns the data and as such, I'd argue that the user should have the right to share the data between different applications however they see fit. However, businesses tend to think that they somehow have the legal (moral even?) right to keep that data in their walled gardens. For as long as this (imo unfair) stance is common, I think that data extraction by use of these anti-bot-bypassing technologies is fair game.
And most scraping isn't done by users. It's done by companies. For profit. Often for less than enlightened reasons.
LinkedIn is a good example: I want my data displayed to people on that job site. I don't want it harvested by every recruiter under the sun who will then spam me. I certainly don't want that data sold between those recruiters long after I deleted my account on LinkedIn. Tinder and sites like that are also an obvious example: yes it's (semi-)public, but I also wouldn't want it to be scraped and harvested by some company – I just want it to be shown temporarily to a limited set of people.
And, in general, I take the fact that you published something on the Internet as a tacit moral consentment for the rest of the world to use it how they want.
This comes with a couple of big asterisks, because (1) Copyright law exists, and I generally try to not break the law, even if I don't agree with it. But the discussion in this thread is mostly separate from copyright: for instance, I don't think a court would see someone scraping and redistributing data from someone's LinkedIn profile as a copyright infringement case.
And (2) because I think that in some specific cases, using published data can be morally wrong, but not as a general rule.
i feel the 'antibot' stuff is more related to the adtech industry vs site-scrapers - remember getting a dedicated server and having friends click on links just to pay for it? Geocities and all these free websites, the biggest costs were bandwidth and storage (not that its not now)
since the AI Boom, there's just more hype over people wanting 'credit' (or money) for something they posted on a forum X-units of time ago.
its called the World Wide Web for a reason.. keep it open, even if it is to 'a bot' - never know when somebody's 'bot software' is reading your webpage for somebody who has some disadvantage and needs assistance
The best you can do is try to form your own little community in a corner of the internet - which is actually a great idea. Mass social media has so many problems - trying to make a little nook for yourself and a small group of enthusiasts/friends seems like the best way to go.
Quite a few things would be vastly better when the information that is on websites is not just publicly readable but publicised in the full sense of the word. Several industries are paying 20~40% revenue just to be findable.
Now maybe scrapers aren't the best solution, but making stuff less accessible is usually not doing much good. Especially not for accessibility.
I suspect that over the next 100 years, as AI and biotech come together, the boundaries between human and bot will start to blur. At some point it will be unethical to differentiate between human, bot, and everything in-between.
I argue that that time is now, if at the very least for the sake of AI-based assistive technologies to be developed to turn a disabled {100% human} into a highly functional {80% human + 20% bot}.
Bot detection should, for this reason, be doomed as an ADA violation.
What if all users of the www were treated equally. There is in practice a conflict of interest between www users calling themselves "developers" and the rest of www users. Yet these "developers" purport to act 100% on behalf of the rest of www users, not themselves. The failure to acknowledge any self-interest is dishonest.
If you really really really want a technical mitigation in addition to a legal one, then paywalls are vastly better than incredibly messy and problematic heuristic bot-detection. There's a huge payment problem for content anyway, so paywalls would mitigate or solve many other problems in addition to the AI-training-data one.
On the other hand, there are a lot of people right now that want to keep stuff accessible to humans and not have it scraped for models. I know the lid's completely off the box on that front so it's probably useless to fight it, but seeing products explicitly designed to circumvent bot prevention feels kinda bad.