Re:“Stop reverse engineering our code”
by discreditable on Tuesday August 11, 2015 @08:02AM
(#10040346)
Link is giving me a 404. Anyone got a mirror?
[ Parent
| on HN ]
Re:“Stop reverse engineering our code”
by anglebracket on Tuesday August 11, 2015 @08:04AM
(#10040354)
Just in case a robot.txt kills that
It's also on seclists.org -
Would archive.org typically honor a robots.txt for a resource it already retrieved? I never understood the intent of a robots.txt to be retroactive.
Apparently yes, it would: https://archive.org/about/exclude.php
Re:“Stop reverse engineering our code”
by syncsynchalt on Tuesday August 11, 2015 @05:23PM
(#10044493)
My understanding is that sites like archive.org honor robots.txt retroactively not because they are required to, but to best honor the wishes of the content provider.