He wants to provide a free service, which he is not obligated to do and costs money for him to do, for individuals to check their email. He never intended for bots to check a billion emails, and obviously doesn't want to pay for that.
That should be respected, and people failing to respect it is why we see the destruction of the open web with things like remote attestation as the only way forward.
Complaints like "but the open web" or "but my exotic browser" are honestly worth nothing against a potential solution for a real, pressing issue like spam requests and bots - if we want to keep the nice things we better start coming up with alternative solutions to the real problem, because corporations will decide the future for us if we don't. Ignoring it will not work out for us.
Web services are intended for humans to use them, and all abuse also ultimately comes from humans directing computers to be abusive. Thus, rather than attaching a computer's temporary identity (an IP address) to the request, we should be attaching a human identity to it. Note here, that I don't care whether the request comes from John Smith in New York. I care about being able to ban you from the service if you are abusive, no matter how many computers you have at your disposal now or in the future.
There's lots of downsides I haven't got solutions for, such as, how do we stop sites cross referencing to reliably discover all users real identities (Google analytics would love this!), and so much more. But we'll have to give up something - if we do nothing we give up everything, maybe giving up only absolute anonymity could be preferred.
You can have a group of people sharing a block and collectively responding to abuse reports to slightly improve privacy. That at least shields you somewhat from the big tech firms.
Essentially a large number of virtual private micro-ISPs.
disclaimer: acquaintance works in the spam business. someone i tried to steer clear of, but was fascinated of what they told me.
IP reputation is a huge thing in the spam world. they pay top dollar for residential US/UK/etc IPs which they can then use for spamming others. and we're not talking one or two IPs. we're talking 100s of thousands of IPs being transacted daily globally. all for spam.
for anyone interested in seeing how low we've come, they have a huge convention in Las Vegas. one should visit it to better understand a field that has been growing like crazy but so few know it. apparently everyone is preparing for a big boom next year, with 10s of $B ready to be deployed.
The point is to not get hung up on the client as a security boundary (it isn't, can't be, won't be), but to focus on the actual harm -- excessive use of the limited resources provided.
And you have to frame your security posture as rooted in the server side throttling, heuristics, et cetera. Flipping out because the client isn't what you expected isn't going to help (determined attackers can look like "vanilla" clients), and is just going to harm the long tail of actual users who are not bots.
I mean, it's built into Safari and Cloudflare supposedly uses it for its CAPTCHAs: https://developer.apple.com/news/?id=huqjyh7k
It's not as invasive as Google's attempt to circumvent ad blockers, but it's still a remote attestation system implemented in the wild already.
Is it really built into Safari, or are you referring to the iCloud option that "privately" helps reduce Cloudflare demands?
The two companies are working together to make this an official web standard, but I haven't heard about it for a while. Maybe they're just laying low after seeing the blowback on Google's (worse) attempts at attesting devices.
Indeed. To state the obvious: the real problem are bad actors - no matter if they are nation states, cybercriminals or people running compromised devices - and their accomplices such as ISPs not responding to abuse reports.
As long as we don't get that under control (say, by threatening to cut offender countries and ISPs from the Internet and SS7 phone networks) we'll have to continue whack-a-mole'ing.
You're going to have zero horid scraping once that's in place.
Not a good choice, why not make this a hashed db that could be distributed freely? Why are bots meant to be excluded? This is bad UX, I'd want to check my emails periodically in the background, but this "anti-bot" measure is meant to make me unable to do so and then demand payment. So this is openly a fight.
> are honestly worth nothing against a potential solution for a real, pressing issue like spam requests and bots
That is not an issue for me at all. Ok, let's face it - maybe I'm just on the other side. For me scrapers are very useful because they reduce the price needed to access the data as they introduce competition. For example price comparison sites are very useful.
The other thing about "waiting" is that bots may not want to do that, or maybe some sort of deadline is sooner than such waiting would allow.
To me, requiring a unique key to be input with the search, that is created after X amount of time (both provided by the initial response and increasing exponentially for subsequent requests) seems like it could be sufficient. If the next request is sooner than X then block them for Y amount of time for attempting to bypass allowable behavior. Allow like 5-10 emails then implement the wait-based functionality so that most non-bots would be fine. After all we're talking about blocking an endpoint only ever meant to be used via the website by actual users, typically they are not trying to check thousands of emails super fast.
Programmatic? Yes. Against their wishes? Also, yes.
The guilt or bad feelings subsided after realizing I'll use more of their bandwidth how I was searching before via the browser. Using the API, I see when the last inventory update occurred, and only after this will I search their discount inventory.
(In any case, if they are on EC2, I cost them 3 to 8 cents per month, estimating high, which they have certainly recouped.)
They could (incorrectly) call you a "bot" but that's simply not true. You are human.
The amount of server resources you deplete is actually less than a modern graphical browser from an ad-sponsored team of software developers.
However perhaps you are not looking at some ads or submitting yourself to tracking or telemetry. Under the W3C web ethics principles you can consume the data from the web in any way you like.
It is your right to decide what HTTP requests you make. XHR/fetch in someone else's Javascript could be well-intentioned but too often it tries to take away some of that agency and transfer it to a web developer.
For tracking, they'll have to rely on old-fashioned "order history" and "inventory period". By these metrics, I would be baffled if they decided to block my access or close my account, even if I'm bypassing Google Tags/their AB test suite/social media analytics/ads for their other services.
Ignoring their nonconsent feels impolite, but they can dry their tears with my spent money.
I've seen many games die due to bots. If developers just straight-up allow them all the normal players quit.
> 2.12 People should be able to render web content as they want
Yes, the choice between Chrome/FF/Safari/Lynx or some other remains
Any discussion not discussing how to deal with abusers is moot
Web developers use JavaScripts to make HTTP requests to API endpoints. The data is being consumed by a script, programmatically. Unless Javascripts are neither scripts nor programs. Good luck with that argument.
There is a W3C TAG Ethical Web Principle that states web users can conusme and display data from the web any way they like. They do not need to use, for example, a particular software or a particular web page design.
2.12 People should be able to render web content as they want
People must be able to change web pages according to their needs. For example, people should be able to install style sheets, assistive browser extensions, and blockers of unwanted content or scripts or auto-played videos. We will build features and write specifications that respect peoples' agency, and will create user agents to represent those preferences on the web user's behalf.
With respect to the JavaScripts authored by web developers and inlined/sourced in web pages, web users have no control over them short of blocking them outright. As such, arguably they are not ideal for making HTTP requests to API endpoints. Unfortunately these scripts can be, and are, used to deny web users' agency.