Re:Analyzing One Million Robots.txt Files
by toomuchtodo on Sunday December 24, 2017 @04:04PM
(#16001290)
I specifically go after sites for archiving that block the Internet Archive in their robots.txt file.
The Internet Archive is also modifying its policy on retroactive blocking using robots.txt, although I don’t have the blog post link handy at the moment.
If you’d like to mirror certain Internet Archive contents, every item is served as a torrent.
The Internet Archive is also modifying its policy on retroactive blocking using robots.txt, although I don’t have the blog post link handy at the moment.
If you’d like to mirror certain Internet Archive contents, every item is served as a torrent.