It's also on seclists.org -
Would archive.org typically honor a robots.txt for a resource it already retrieved? I never understood the intent of a robots.txt to be retroactive.
Apparently yes, it would: https://archive.org/about/exclude.php
Re:“Stop reverse engineering our code”
by syncsynchalt on Tuesday August 11, 2015 @05:23PM
(#10044493)
My understanding is that sites like archive.org honor robots.txt retroactively not because they are required to, but to best honor the wishes of the content provider.
http://pastebin.com/rcPSyRnR