News for hackers. Stuff that matters.
Re:Cloudflare Introduces Default Blocking of A.I. Data Scrapers by dceddia on Wednesday July 02, 2025 @02:29PM (#44447199)
Yep this terrifies me, 100%. We’re slowly losing the open internet and the frog is being boiled slowly enough that people are very happy to defend the rising temperature.

If DDoS wasn’t a scary enough boogeyman to get people to install Cloudflare as a man-in-the-middle on all their website traffic, maybe the threat of AI scrapers will do the trick?

The thing about this slow slide is it’s always defensible. Someone can always say “but I don’t want my site to be scraped, and this service is free, or even better yet, I can set up my own toll booth and collect money! They’re wonderful!”

Trouble is, one day, at this rate, almost all internet traffic will be going through that same gate. And once they have literally everyone (and all their traffic)… well, internet access is an immense amount of power to wield and I can’t see a world in which it remains untainted by commercial and government interests forever.

And “forever” is what’s at stake, because it’ll be near impossible to recover from once 99% of the population is happy to use one of the 3 approved browsers on the 2 approved devices (latest version only). Feels like we’re already accepting that future at an increasing rate.

Re:Cloudflare Introduces Default Blocking of A.I. Data Scrapers by mattl on Wednesday July 02, 2025 @04:16PM (#44448321)
We need a reasonable alternative to some of what Cloudflare does that can be easily installed as a package on Linux distributions without any of the following to install it.

* curl | bash * Docker * Anything that smacks of cryptocurrency or other scams

Just a standard repo for Debian and RHEL derived distros. Fully open source so everyone can use it.

Until that exists, using Cloudflare is inevitable.

It needs to be able to at least:

* provide some basic security (something to check for sql injection, etc) * rate limiting * User agent blocking * IP address and ASN blocking

Make it easy to set up with sensible defaults and a way to subscribe to blocklists.

Re:Cloudflare Introduces Default Blocking of A.I. Data Scrapers by saint_yossarian on Wednesday July 02, 2025 @06:40PM (#44449597)
I remember using mod_security with Apache long ago for some of this, looks like it's still around and now also supports Nginx and IIS: https://modsecurity.org/
Re:Cloudflare Introduces Default Blocking of A.I. Data Scrapers by mattl on Wednesday July 02, 2025 @08:56PM (#44450510)
Thank you. This doesn't have everything I'm looking for, but apparently it has been packaged in Debian at least. I don't know why the website doesn't mention this.
Re:Cloudflare Introduces Default Blocking of A.I. Data Scrapers by 1oooqooq on Wednesday July 02, 2025 @07:41PM (#44450022)
it's called not having a vibecoded app that falls to pieces on public endpoints even before ngix ratelimit can kick in
Re:Cloudflare Introduces Default Blocking of A.I. Data Scrapers by mattl on Wednesday July 02, 2025 @08:56PM (#44450502)
Nobody is talking about a vibe coded app. I want to block AI scrapers entirely.
Re:Cloudflare Introduces Default Blocking of A.I. Data Scrapers by 1oooqooq on Friday July 04, 2025 @07:11AM (#44463457)
point is, why do you care if your site can handle the traffic?

there's no (malicious) bot detection that won't impact a portion of real users. accept that fact and just let it be.

poisoning data in ways that's obvious to the false positive user is a much better option.

Re:Cloudflare Introduces Default Blocking of A.I. Data Scrapers by mattl on Friday July 04, 2025 @11:38AM (#44465468)
I really doubt any legit user is using a weird user agent and an IP address in the same AS as an AI slop crawler
Re:Cloudflare Introduces Default Blocking of A.I. Data Scrapers by 1oooqooq on Friday July 04, 2025 @03:40PM (#44467284)
You'd be surprised. Your users too, but you wouldn't know because they will not be able to tell you.
Re:Cloudflare Introduces Default Blocking of A.I. Data Scrapers by xena on Wednesday July 02, 2025 @08:43PM (#44450414)
I make this: https://anubis.techaro.lol. I have yet to add the SQL injection or IP list layers, but I can add that to the roadmap.
Re:Cloudflare Introduces Default Blocking of A.I. Data Scrapers by mattl on Wednesday July 02, 2025 @08:55PM (#44450498)
The proof of work stuff feels so cryptocurrency adjacent that I've been looking at other tools for my own thing, but I've seen Anubis on other websites and it seems to do a good job.
Re:Cloudflare Introduces Default Blocking of A.I. Data Scrapers by xena on Wednesday July 02, 2025 @09:42PM (#44450790)
There's a non proof of work challenge: https://anubis.techaro.lol/docs/admin/configuration/challeng...

Also: Anubis does not mine cryptocurrency. Proof of work is easy to validate on the server and economically scales poorly in the wild for abusive scrapers.

Re:Cloudflare Introduces Default Blocking of A.I. Data Scrapers by mattl on Wednesday July 02, 2025 @11:28PM (#44451371)
Thanks for the link. I’ll have a look.

I’m glad there’s no cryptocurrency involved (was never a concern) but I worry about the optics of something so closely associated.

(I appreciate your commenting on this. I know the project recently blew up in popularity. Keep up the great work)

Re:Cloudflare Introduces Default Blocking of A.I. Data Scrapers by xena on Thursday July 03, 2025 @12:52AM (#44451725)
If you have suggestions for JS based challenges that don't become a case of "read the source code to figure out how to make playwright lie", I'm all ears for the ideas :)
Re:Cloudflare Introduces Default Blocking of A.I. Data Scrapers by fsflover on Thursday July 03, 2025 @03:40AM (#44452596)
This unsubstantiated anti-cryptocurrency bias on HN is quite disappointing. Did you hear about filecoin, which allows to buy and sell disk space independently on large companies? Why wouldn't an anonymous cryptocurrency like Monero help with this real problem? What would the downsides be?
Re:Cloudflare Introduces Default Blocking of A.I. Data Scrapers by v5v3 on Thursday July 03, 2025 @07:27AM (#44453893)
Primary reason people use cloudflare is to hide the ip address of their own server. So they are less likely to be hacked.

Most people are not worried about DDos as their is no reason for any one to DDos them.

Until other services start offering the same, Cloudflare remains default.

Re:Cloudflare Introduces Default Blocking of A.I. Data Scrapers by RiverCrochet on Wednesday July 02, 2025 @02:57PM (#44447485)
The Internet is not the first global network. Before the Internet, you had the global telephone network. It, too, strangulated end users, but eventually became stagnant, overpriced, and irrelevant. Super long-term, the current Internet is not immune from this. Internet standards are about getting as complicated and quirky as the old Bell stuff that was trying to make miles of buried copper the future, and if regulatory/commercial forces freeze this stuff in place, it's going to lead to stagnation eventually.

Something coming down the pike I think, for example, is that IPv4 addresses are going to get realllly expensive soon. That's going to lead to all sorts of interesting things in the Internet landscape and their applications.

I'm sure we'll probably have to spend some decades in the "approved devices and browers only" world before a next wave comes.