Re:Only Google is really allowed to crawl the web
by luckylion on Saturday March 27, 2021 @04:53AM
(#26600966)
From some I could, but why would I? If they're not adding value and they don't want to behave, I don't see a reason to spend money to adapt my systems to be "inclusive" towards their usage patterns.
Re:Only Google is really allowed to crawl the web
by d110af5ccf on Sunday March 28, 2021 @02:13AM
(#26608383)
In context, you're justifying blocking all automated traffic, even that which does behave, by pointing out that some of it doesn't. That attitude seems lazy at best, malicious at worst.
Re:Only Google is really allowed to crawl the web
by MinorTom on Friday March 26, 2021 @08:10PM
(#26598503)
CGNAT
Re:Only Google is really allowed to crawl the web
by d110af5ccf on Sunday March 28, 2021 @02:09AM
(#26608373)
Now that's a really good point. I wonder why there isn't a standard protocol for signalling upstream that a particular connection is abusive and to please rate limit the path at the source on your behalf? It would certainly add complexity, but the current situation is hardly better.
Why can't you just ratelimit IPs that are "too active" for your server to handle?