News for hackers. Stuff that matters.

A comment on Only Google is really allowed to crawl the web. Parent comment

Re:Only Google is really allowed to crawl the web by the_dege on Friday March 26, 2021 @12:54PM (#26594298)
Sometimes website admins will also try to report your ips to the service provider as a source of attacks (even if not true).
[ Parent | on HN ]
Re:Only Google is really allowed to crawl the web by DocTomoe on Friday March 26, 2021 @05:37PM (#26597258)
Given how often I've had misbehaving crawlers slow own servers in the early 2000s, I do not see how a crawler that disobeys robots.txt is not an attempted attack.
[ Parent | on HN ]