But Google do allow an opt-out and follows what AT consider outdated methods of data protection. Doesn't seem that AT is the same.
Re:Robots.txt is a suicide note
by stonemetal on Tuesday May 10, 2011 @12:00PM
(#2532939)
The old it isn't illegal because bob and joe do it too defense. The behavior of others don't change the legality of your behavior.
I don't know if AT does or does not do anything illegal but what you have outlined is not illegal per se. archive.org and Google do it too.