News for hackers. Stuff that matters.
Re:“Stop reverse engineering our code” by hughw on Tuesday August 11, 2015 @10:40AM (#10041376)
Would archive.org typically honor a robots.txt for a resource it already retrieved? I never understood the intent of a robots.txt to be retroactive.
Re:“Stop reverse engineering our code” by mikeash on Tuesday August 11, 2015 @01:46PM (#10042985)
Apparently yes, it would: https://archive.org/about/exclude.php
Re:“Stop reverse engineering our code” by syncsynchalt on Tuesday August 11, 2015 @05:23PM (#10044493)
My understanding is that sites like archive.org honor robots.txt retroactively not because they are required to, but to best honor the wishes of the content provider.
Re:“Stop reverse engineering our code” by X-Istence on Tuesday August 11, 2015 @03:50PM (#10043843)
Yes, it simply hides the content, it is still kept in their database so if the robots.txt disappears, it pops back from their archive.

New pages won't be archived though.