I remember using mod_security with Apache long ago for some of this, looks like it's still around and now also supports Nginx and IIS: https://modsecurity.org/

Thank you. This doesn't have everything I'm looking for, but apparently it has been packaged in Debian at least. I don't know why the website doesn't mention this.

it's called not having a vibecoded app that falls to pieces on public endpoints even before ngix ratelimit can kick in

Nobody is talking about a vibe coded app. I want to block AI scrapers entirely.

point is, why do you care if your site can handle the traffic?

there's no (malicious) bot detection that won't impact a portion of real users. accept that fact and just let it be.

poisoning data in ways that's obvious to the false positive user is a much better option.

I really doubt any legit user is using a weird user agent and an IP address in the same AS as an AI slop crawler

You'd be surprised. Your users too, but you wouldn't know because they will not be able to tell you.

I make this: https://anubis.techaro.lol. I have yet to add the SQL injection or IP list layers, but I can add that to the roadmap.

The proof of work stuff feels so cryptocurrency adjacent that I've been looking at other tools for my own thing, but I've seen Anubis on other websites and it seems to do a good job.

There's a non proof of work challenge: https://anubis.techaro.lol/docs/admin/configuration/challeng...

Also: Anubis does not mine cryptocurrency. Proof of work is easy to validate on the server and economically scales poorly in the wild for abusive scrapers.

Thanks for the link. I’ll have a look.

I’m glad there’s no cryptocurrency involved (was never a concern) but I worry about the optics of something so closely associated.

(I appreciate your commenting on this. I know the project recently blew up in popularity. Keep up the great work)

If you have suggestions for JS based challenges that don't become a case of "read the source code to figure out how to make playwright lie", I'm all ears for the ideas :)

This unsubstantiated anti-cryptocurrency bias on HN is quite disappointing. Did you hear about filecoin, which allows to buy and sell disk space independently on large companies? Why wouldn't an anonymous cryptocurrency like Monero help with this real problem? What would the downsides be?

Primary reason people use cloudflare is to hide the ip address of their own server. So they are less likely to be hacked.

Most people are not worried about DDos as their is no reason for any one to DDos them.

Until other services start offering the same, Cloudflare remains default.