What one may find in robots.txt

Re:What one may find in robots.txt by neil_s on Monday May 18, 2015 @10:05AM (#9564458)

Wow, all the US Department of State files have just gone missing from archive.org. The servers hosting those files are conveniently down.

[ Parent | on HN ]

Re:What one may find in robots.txt by benyami on Monday May 18, 2015 @10:53AM (#9564766)

Check out the Internet Archive FAQ on how to remove a document from their archives. https://archive.org/about/exclude.php

It looks like they used robots.txt to do that.

[ Parent | on HN ]

Re:What one may find in robots.txt by neil_s on Monday May 18, 2015 @10:58AM (#9564798)

Huh, so the wild-card user-agent will block not just searchbots, but also archivebots. Wonder how OP managed to get screenshots of archive.org having archives available for those documents.

[ Parent | on HN ]

Re:What one may find in robots.txt by kjell on Monday May 18, 2015 @03:49PM (#9566482)

They're there, at least the two I looked at.

https://web.archive.org/web/20130413152316/http://www.state....

Each line is missing `/documents` in the snippet of the `robots.txt`

[ Parent | on HN ]

Re:What one may find in robots.txt by maxmcd on Monday May 18, 2015 @12:14PM (#9565226)

I have been able to view multiple pdfs and view the page screenshotted by the author.

[ Parent | on HN ]