News for hackers. Stuff that matters.
Re:Only Google is really allowed to crawl the web by d110af5ccf on Friday March 26, 2021 @07:09PM (#26598084)
> they're hammering your server

Why can't you just ratelimit IPs that are "too active" for your server to handle?

Re:Only Google is really allowed to crawl the web by luckylion on Saturday March 27, 2021 @04:53AM (#26600966)
From some I could, but why would I? If they're not adding value and they don't want to behave, I don't see a reason to spend money to adapt my systems to be "inclusive" towards their usage patterns.
Re:Only Google is really allowed to crawl the web by d110af5ccf on Sunday March 28, 2021 @02:13AM (#26608383)
In context, you're justifying blocking all automated traffic, even that which does behave, by pointing out that some of it doesn't. That attitude seems lazy at best, malicious at worst.
Re:Only Google is really allowed to crawl the web by MinorTom on Friday March 26, 2021 @08:10PM (#26598503)
CGNAT
Re:Only Google is really allowed to crawl the web by d110af5ccf on Sunday March 28, 2021 @02:09AM (#26608373)
Now that's a really good point. I wonder why there isn't a standard protocol for signalling upstream that a particular connection is abusive and to please rate limit the path at the source on your behalf? It would certainly add complexity, but the current situation is hardly better.