News for hackers. Stuff that matters.
Re:Handling bot attacks against a Tor hidden service by stefantalpalaru on Sunday May 07, 2017 @08:44AM (#14285174)
> I'd say not holding to the standards of robots.txt and 403-Forbidden is quite malicious, just not evil or bad. If you build a crawler, you should play nice.

https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea... :

> A few months ago we stopped referring to robots.txt files on U.S. government and military web sites for both crawling and displaying web pages (though we respond to removal requests sent to [email protected]). As we have moved towards broader access it has not caused problems, which we take as a good sign. We are now looking to do this more broadly.

There are no comments on this comment yet.