Re:Let us serve you, but don't bring us down
by crazygringo on Monday May 29, 2023 @03:35PM
(#36117181)
But it's precisely those complicated parts though that seem like they should be solved by now. The solution is both of the problems is for when traffic gets unexpectedly high either behind a NAT or globally, everybody gets a CATPCHA when gives you a token and then you continue to rate-limit each token. (If users log in then each login already does this, no CAPTCHA needed.) This strategy is basically what e.g. CloudFlare does with the CAPTCHAs.
Obviously if someone is attempting a large-scale DDoS your servers can't handle it and you'll be using CloudFlare for its scale. But otherwise, for basic protection against greedy spiders who are even trying to evade detection across a ton of VPN/cloud IP's, this strategy works fine. It's exactly the kind of thing that I would expect any large website to implement.
If there isn't an open-source tool that does this, I wonder why not. Or if there is, I wonder why IA isn't using something like it. But heck, IA wasn't even using a simple version -- it was just 64 IP addresses where basic rate-limiting would have worked fine.
Obviously if someone is attempting a large-scale DDoS your servers can't handle it and you'll be using CloudFlare for its scale. But otherwise, for basic protection against greedy spiders who are even trying to evade detection across a ton of VPN/cloud IP's, this strategy works fine. It's exactly the kind of thing that I would expect any large website to implement.
If there isn't an open-source tool that does this, I wonder why not. Or if there is, I wonder why IA isn't using something like it. But heck, IA wasn't even using a simple version -- it was just 64 IP addresses where basic rate-limiting would have worked fine.