Re:“Stop reverse engineering our code”
by hughw on Tuesday August 11, 2015 @10:40AM
(#10041376)
Would archive.org typically honor a robots.txt for a resource it already retrieved? I never understood the intent of a robots.txt to be retroactive.
[ Parent
| on HN ]
Apparently yes, it would: https://archive.org/about/exclude.php
Re:“Stop reverse engineering our code”
by syncsynchalt on Tuesday August 11, 2015 @05:23PM
(#10044493)
My understanding is that sites like archive.org honor robots.txt retroactively not because they are required to, but to best honor the wishes of the content provider.