Re:“Stop reverse engineering our code”
by syncsynchalt on Tuesday August 11, 2015 @05:23PM
(#10044493)
My understanding is that sites like archive.org honor robots.txt retroactively not because they are required to, but to best honor the wishes of the content provider.
[ Parent
| on HN ]
There are no comments on this comment yet.