News for hackers. Stuff that matters.
Re:Analyzing One Million Robots.txt Files by toomuchtodo on Sunday December 24, 2017 @04:04PM (#16001290)
I specifically go after sites for archiving that block the Internet Archive in their robots.txt file.

The Internet Archive is also modifying its policy on retroactive blocking using robots.txt, although I don’t have the blog post link handy at the moment.

If you’d like to mirror certain Internet Archive contents, every item is served as a torrent.

Re:Analyzing One Million Robots.txt Files by mynewtb on Sunday December 24, 2017 @05:40PM (#16001695)
The Wayback Machine data is not available in bulk, only via the web interface.