News for hackers. Stuff that matters.

A comment on Only Google is really allowed to crawl the web. Parent comment

Re:Only Google is really allowed to crawl the web by luckylion on Saturday March 27, 2021 @04:53AM (#26600966)
From some I could, but why would I? If they're not adding value and they don't want to behave, I don't see a reason to spend money to adapt my systems to be "inclusive" towards their usage patterns.
[ Parent | on HN ]
Re:Only Google is really allowed to crawl the web by d110af5ccf on Sunday March 28, 2021 @02:13AM (#26608383)
In context, you're justifying blocking all automated traffic, even that which does behave, by pointing out that some of it doesn't. That attitude seems lazy at best, malicious at worst.
[ Parent | on HN ]