News for hackers. Stuff that matters.

A comment on Only Google is really allowed to crawl the web. Parent comment

Re:Only Google is really allowed to crawl the web by sn_master on Friday March 26, 2021 @11:00PM (#26599525)
Their IP range can be blocked. Google has known IP ranges and if the website admin wants can allow them based on that rather than robots.txt, and that would really mess up everyone else.
[ Parent | on HN ]

There are no comments on this comment yet.