News for hackers. Stuff that matters.

A comment on What one may find in robots.txt. Parent comment

Re:What one may find in robots.txt by benyami on Monday May 18, 2015 @10:53AM (#9564766)
Check out the Internet Archive FAQ on how to remove a document from their archives. https://archive.org/about/exclude.php

It looks like they used robots.txt to do that.

[ Parent | on HN ]
Re:What one may find in robots.txt by neil_s on Monday May 18, 2015 @10:58AM (#9564798)
Huh, so the wild-card user-agent will block not just searchbots, but also archivebots. Wonder how OP managed to get screenshots of archive.org having archives available for those documents.
[ Parent | on HN ]