There are residential-IP-backed VPN services that you can use just like commercial VPN services — but they're mostly built on the backs of botnets, so it's ethically questionable to use them.
They are easily detected if you are buying IP intelligence from one of the higher quality providers: https://app.spur.us/context?q=STARVPN_PROXY
CloudFront being way past the simple blocking of IP addresses, I wouldn't be surprised if a mismatch between your IP block and your language/cookies would be enough to lower your score.
It’s only easy to bypass if you’re scraping or doing nefarious stuff.
The old IP address was a mom-and-pop CGNAT.
Thanks CF, for protecting us from capitalism, I guess?
"We have a problem with bots" - "Just create a firewall rule, whatever"
But the immediate response to bots shouldn't be "make everyone go through a captcha". There's lots of nuance that you can tune to deal with your particular situation, but the first thing I'd do is block known bots or ASNs, set up a limit to trigger (bots usually don't make 1 document request a minute), set up higher limits for users who (seem to) have a valid cookie indicating that they are logged in, set up different thresholds for certain countries that are more risky etc etc.
What you need to protect your service depends on your situation, it's not a one-size-fits-all solution. E.g. I find that I have no automated contact form spam once I add a simple JS to add some data that isn't standard, but I'm sure that wouldn't hold up if there was enough incentive to try to get past it.
But the OP mentioned not just free services, but e.g. webhosting logins. That's just sad, as is Cloudflare's community being behind an aggressive captcha. I'm a user, I'm logged in, I've posted before, I'm in good standing, yet when I go there, I need to solve a captcha. When I then go there again an hour later, guess what, another captcha.
Either there's another reason I'm not seeing or it's just lazyness as in "we need to have a forum but we really don't want to spend any resources on it, just put up an aggressive captcha that'll filter out most bots and everyone but the determined users".
> I'm a user, I'm logged in, I've posted before, I'm in good standing, yet when I go there, I need to solve a captcha.
Though consider the fact that taking over someone's account shouldn't give you (a spammer) unlimited access either. The spambots you see on Twitter are mostly cred-stuffed accounts. It's a hard problem. Existing accounts are more dangerous than fresh accounts.
Imo, "write your own password" should be a thing of the past. Services should just auto-gen a password or there should be a way to require the OS (like a password manager) to generate one to avoid cred-stuffing. We're letting down the average person by making them come up with unique passwords for every service instead of just helping them. Though I'm way off topic.
But it's not unlimited access -- it's _read_ access at that point. This is just when trying to access the forums at all, not when trying to post a message. And if they were worried about evildoers scraping all the data from their forums, they could rate-limit and then require captchas (their WAF settings make that trivial). But they don't, or the rate limiting is so generous that I've never hit it, and their forums are not that active, so I don't think that's the reason.
Adding more protection to an endpoint where users send posts makes some sense, but for reading? On their dashboard you need to solve the captcha on the login-form. On the forums, you cannot even get to the login (which works via the dashboard, where you'll solve a captcha again) until you've solved the captcha.
I use and like CF's products a lot (I'm a paying customer, I'm not even looking for free support on the forums, but their docs are lacking a lot of information that I'm interested in), so I don't believe in "we're incompetent", keeping the resource-investment low by filtering out bots and a chunk of users makes a lot more sense.
That's not correct, Cloudflare challenge pages / Turnstile will never show you a puzzle.
Anyway, I know the "Cloudflare's monopoly gating is killing web openness!" meme is common online, especially on HN, but in real life I've never actually heard anyone else complain about it (either a fellow dev or a customer or a manager). Instead, it's been universal praise for the actual issues Cloudflare exists to solve (CDN, bot protection, serverless, etc)... they are a godsend for small businesses that otherwise get immediately flooded by spam requests, especially from China, Russia, and India.
And if you think Cloudflare is bad, it was even worse before they became dominant, with terrible services like Incapsula/Imperva charging way more but providing both worse bot protection AND more false positives, or the really hard early reCAPTCHAs (that Cloudflare was largely able to replace, for users who DO fit within the "norm"). That, or you'd have to fight every random sysadmin with their own lazy rules, like firewall rules that blacklisted entire regional ISPs and took weeks or months to resolve, if they ever even checked their emails.
As inconvenient as Cloudflare is for users who take privacy seriously and try to be less trackable, for the other 90% of us who don't care as much and easily fit into their "norm" model, it's much nicer than what came before. Site downtime and slowness are also much less common now, in no small part because of their easy CDN and caching.
From the implementation side, I've set up a few Cloudflare accounts in my career, but do take the time to try to configure it to balance security vs accessibility for any given target audience. Sometimes we'd block entire countries, other times we'd minimize security to ensure maximum reach, but usually we'd customize rulesets in the middle for any given company & audience. I never got a complaint about it (our emails were still available and not blocked).
This was always a direct response to some business need, usually spambots or DDoS attempts that fail2ban etc. couldn't catch well enough. For the business, it was usually a "shit, our website is down again, what is it this time", and the choice between "for free or $20 we can get it back up again and not have this issue anymore" or "we can spend thousands of dollars and weeks of labor building our own security solution" is pretty easy. "What about that one guy who is proxied behind TOR and three VPNs with a random user agent using a text-only browser he wrote himself?" never really factors into that process =/ There's just not enough users like that out in the wild vs the very real constant threat of bots and malware.
It's a shitty situation that the web is like this today, and I wish it weren't the case, but it really is an arms race, and these imperfect weapons are just what most of us have access to...
On my small website, bot traffic is almost entirely from DigitalOcean VPSs.
Maybe in your country, but tons of countries outside of the US (first world) avoid Macs like the plague and just use Linux/Windows as building machines.
But you are right on Google/Cloudflare, they are the poison of the web.
Some non-existant system of attesting that I'm person X (possibly through an e-ID card) who has issued a client certificate Y (cert chain, using my e-ID cert to sign) to be used with my device Z (presumably with a device fingerprint or IP range attached to the cert). Of course, this would mean no privacy, but that's not that different from being signed in through Google as an identity provider, we'd just shift the mechanism to be universal (like client certs already are). One of the options that would take more coordination than will probably happen (though very similar to some e-signature solutions in EU, which we already use) but I could see using something like that for a variety of professional/service sites, since signing in with the e-ID card directly is already a thing on some sites here (government sites, banking sites, utilities sites).
I had a guy like that working with me. Blocked every possible tracker, disabled javascript, used some niche browser, proton mail, and then complains that google doesn’t allow him to sign in. I get it, privacy and what not. But the guy was an outlier.
Some random blogs, product pages aren’t gov, most likely have no way to opt-in for gov eID (maybe they aren’t based in the EU), and they only care that their service is available fast globally and that they get ddos protection for free (plus some other convenience features).
We already do a simpler version of that with TLS and HTTPS, there are globally trusted root certs that ship with most OSes and browsers. It's just that we haven't extended the same approach to client certs and identity verification, instead having a bunch of walled gardens and governments running legacy methods of figuring out who someone is, as opposed to various eID mechanisms.
If I trust news.ycombinator.com because I trust ISRG Root X1, I might similarly trust John Doe's iPhone because I trust the government of France's CA, as a hypothetical, as long as the certification chain is valid there.
It's a problem that's technically solvable (say, in 20-50 years), but won't get done because good luck getting a bunch of governments to collaborate on that across the world. It's actually a surprise that we have TLS in the first place.
Hey, by the way, would you trust some Chinese or Russian root certificate?
The question is irrelevant, frankly. Consider this: you’re living in Germany today. You trust the German government. They handle all your logins using that eID. What if in February AfD comes to power? Do you still trust the German government? Governments are formed by people. Different people have different interests.
Another good example of something that’s technically feasible and not that complex, but was made infeasible due to either ignorance or malice, with all of the dark UI patterns and scummy behaviour.
> Hey, by the way, would you trust some Chinese or Russian root certificate?
Most people already do: https://chromium.googlesource.com/chromium/src/+/main/net/da...
For example:
CN=CFCA EV ROOT,O=China Financial Certification Authority,C=CN
CN=GDCA TrustAUTH R5 ROOT,O=GUANG DONG CERTIFICATE AUTHORITY CO.,LTD.,C=CN
CN=UCA Global G2 Root,O=UniTrust,C=CN
CN=UCA Extended Validation Root,O=UniTrust,C=CN
CN=vTrus ECC Root CA,O=iTrusChina Co.,Ltd.,C=CN
CN=vTrus Root CA,O=iTrusChina Co.,Ltd.,C=CN
Plus, the opposite is also viable, if for example the Russian govt. wanted to allow anyone to verify whether particular requests come from their citizens, they might also run their own CA akin to https://www.bleepingcomputer.com/news/security/russia-create... except that the attack vector would change from MitM to fake identities being issued by them as needed (but since the server is the one doing the verification, it might as well drop the CA when desired).
> What if in February AfD comes to power?
Revoking the eID and anything dependent on it would be akin to your passport being taken away.
Essentially the modern day digital equivalent of getting your Google account banned by some bot, if you use that account for auth in a bunch of places.
Fundamentally, that’s no different from the reality that we already face - my regular eID could also be taken away if my own government felt like it, same as with my bank account and other assets.
Client certs themselves are nothing new, same for PKI. It’s a cool technology that could but presently cannot solve the problem of client identity globally, because we just can’t have nice things and order.
Is it? If my eID is used for logging in to my bank and said eID is revoked, I can no longer log in to my bank account. That’s completely different than a locked up passport.
> Essentially the modern day digital equivalent of getting your Google account banned by some bot, if you use that account for auth in a bunch of places.
Use a custom domain, don’t make your kingdom dependent on the gmail.com address.
I don’t know, for me the perfect amount of government oversight is “as little as possible”. There’s zero need for the government to mediate between me and my bank, or some random service provider on the internet.
What you’re describing sounds like a fun technical challenge assuming a perfect world. For example: who decides which countries’ certificates should be revoked? Who decides who is the rogue one? Even that is stretching it too far. Can I simply download a browser without some selected certificates? If the technology is so great, why isn’t it widely adopted today
Those are all rhetorical questions. You don’t have explain PKI to me.
Pretty much the same failure mode, just with different immediacy. No more travel, no more ability to start using new banking services, no more proving identity for becoming employed, pretty much anything that needs you to provide valid governmental ID (ID card or passport) and doesn't accept alternatives.
On the opposite end of that, both those services might accept something like a driver's license and the banking service might allow you to log in with their app, or a similar identity provider as a backup.
> There’s zero need for the government to mediate between me and my bank, or some random service provider on the internet.
Who else should we depend upon for verifying the identity of someone? Because currently it's a hodgepodge, especially when some places treat the equivalent of an SSN as a secret or have other half baked mechanisms, whereas in actuality it's a problem that's been solved far better, the same way how e-signatures work here when a single competent authority implements them well (certs on the e-ID card, you choose what to sign, but there's both data integrity and non-repudiation, a service that everyone integrates with and it is basically treated as a commonplace utility).
> What you’re describing sounds like a fun technical challenge assuming a perfect world. ...
Yeah, that's about it. Have a good one!
There are a whole ton of privacy problems with this. I am happy to demonstrate anonymously that I am not a bot, but a random blogger does not need to know that I am John Doe, a citizen of France with national ID number 12345678.
1) People who anonymize their IP, use Linux, a browser with noscript, etc
2) People who are OK with having a government issued digital id and having to use it to access the internet
...look like, in your opinion?
Binding login interaction to some government issued id…who’s entitled here.
Sounds like throwing a baby out with the bathwater.
Of course accessibility is important - ie screen reader compatibility.
A typical testing matrix in the US would be
- Safari for iOS
- Chrome for desktop and Android
- maybe Safari for desktop or you just tell Mac users to use Chrome
- Firefox if you have the time. But if not, no big deal.
We are definitely not going to test for a highly customized Firefox on Linux running over a VPN.
The issues I have are website pretending to be apps and apps that are SPAs for no reasons.
You can do so when your bottom line is healthy. Otherwise you go out of business. That’s business 101.
Everything else - no.
But if I am using standards and they have an ad blocker that blocks some of the functioning of my site, am I also required to test my site against that?
I'd include _everything_ important in the "yes" category. If I cannot access the customer panel to update settings or notify them of a bug that is affecting me because I'm using Firefox ("works for 95% of users"), they're just not keeping up their end of the contract.
Remember, 95% excludes everything but chromium/webkit-engines.
Every company decides which customers are worth going after.
Might still be a business decision, but it's like saying "we'll drop any emails that indicate a mail client other than apple mail/gmail/outlook".
It's not surprising that the strongest protections always happen on the unsubscribe links, but not on the subscribe-links. That just needs to be fined out of existence, just like "you can order with one click, but you need 50 clicks and a three-hour-conversation to cancel".
Just like other cases, I won’t accept that it’s “just lazy” on the part of big tech companies. They clearly know how to adjust their internal view/reputation of a domain once it starts being used for “misbehaviour” and spam such that they start blocking it.
Thus they could clearly start by not doing so-and, maybe, they’re “really touchy” about domains with no initial “internal score” such that if a new domain pops up and starts spamming people they catch it fast. Its not necessary to break open Internet protocols, though, unless they want the breakage.
Where. It’s global internet we communicate via.
One side of the argument is that Cloudflare places an undue burden. The other side of the argument is that without the CF protections, the service provider doesn't even have reason to believe the request is coming from a human being the law protects.
And before you say "that's their choice," you're the one who is breaking the functionality. Nothing about using a VPN or linux or Firefox creates any problem for TCP/IP or https.
However, while the site creator does have to meet the disabled halfway, the disabled person is responsible for having whatever type of equipment they need to make it work - ie screenreaders
Just do your job right. Not saying you should test some unique Firefox config but at least the default version is to be tested.
Hell, I've seen people here indicating that they just tell desktop Mac users to "install Chrome". Such carelessness is bad for business. Web development sure could raise its bar.
For the longest, Amazon Connect’s - AWS hosted call center software - call flow builder only worked with Chrome.
Even for B2C users, using Chrome is not a deal breaker. If they are okay with using shitty Electron apps, they will be okay with using Chrome for Mac.
I do believe that it is true that many site owners wouldn't care. But I suspect that in the vast majority of cases they don't actually know. Cloudflare probably shows them a nice dashboard about all of these blocked "threats" and they don't know better than to question it.
Unfortunately, I think the Cloudflare challenges are designed to filter out users similar to your profile... once you stray far enough from the norm, it just looks like a bot / suspicious traffic to them. Statistically there's not enough users like you (privacy-conscious Linux users on nonstandard browsers) for them to really care enough to do anything about it. Site owners don't care either since you're usually like 1-2% of users at most, and typically also the same ones who block ads, etc., so they don't mind blocking you... it's sad, but I don't think there is really anything you can do about it except conform. It's an ongoing arms race and you're caught in the middle.