I think rate limiting is the wrong approach. If there’s too much load you need to manage the queue
Re:Let us serve you, but don't bring us down
by crazygringo on Monday May 29, 2023 @03:35PM
(#36117184)
Not against bad actors, like this case.
What does "manage the queue" mean? And how would that be better than dropping requests from the source of the abuse?
It isn't that hard to set up naive rate limiting per ip address. It's a few lines in haproxy, and there is documentation on how to do it. There are a couple of problems that make it more complicated though. The first is that with NATs, you can have a lot of users behind a single IP address, which can result in legitimate requests getting blocked. The second is that, while it can help against a DoS, it doesn't help that much against a DDoS, because the requests are coming from a large number of distinct IP addreesses.
Re:Let us serve you, but don't bring us down
by crazygringo on Monday May 29, 2023 @03:35PM
(#36117181)
But it's precisely those complicated parts though that seem like they should be solved by now. The solution is both of the problems is for when traffic gets unexpectedly high either behind a NAT or globally, everybody gets a CATPCHA when gives you a token and then you continue to rate-limit each token. (If users log in then each login already does this, no CAPTCHA needed.) This strategy is basically what e.g. CloudFlare does with the CAPTCHAs.
Obviously if someone is attempting a large-scale DDoS your servers can't handle it and you'll be using CloudFlare for its scale. But otherwise, for basic protection against greedy spiders who are even trying to evade detection across a ton of VPN/cloud IP's, this strategy works fine. It's exactly the kind of thing that I would expect any large website to implement.
If there isn't an open-source tool that does this, I wonder why not. Or if there is, I wonder why IA isn't using something like it. But heck, IA wasn't even using a simple version -- it was just 64 IP addresses where basic rate-limiting would have worked fine.
ipv6 would solve these issues because they remove the need for NATs
And create a new problem: banning by IPv6 is a nightmare https://adam-p.ca/blog/2022/02/ipv6-rate-limiting/
It just strikes me as surprising that sites are still dealing with these problems manually in 2023.
The idea that a site is still manually having to identify a set of IP addresses and block them with human intervention seems absolutely archaic by this point.
And I don't think different sites have particularly different needs here... basic pattern-matching heuristics can identify IP addresses/blocks (plus things like HTTP headers) that are suddenly ramping up requests, and use CAPTCHAs to allow legitimate users through when one IP address is doing NAT for many users behind it. Really the main choice is just whether to block spidering as much as possible, always allow it as long as it's at a reasonable speed, or limit it to a standard whitelist of known orgs (Google, Bing, Facebook, Internet Archive, etc.).
It just strikes me as odd that when you follow a basic tutorial for installing Apache on Linux on Digital Ocean, a rate-limiter isn't part of it. It seems like it should be almost as basic as an HTTPS certificate by this point.