Re:Robots.txt is a suicide note
by sbierwagen on Wednesday May 11, 2011 @02:59PM
(#2537858)
Well, when I iterated through the everything2.com namespace and downloaded the majority of their content[1], I respected their robots.txt. After I did that, they changed it to:
1: http://bbot.org/blog/archives/2011/01/17/more_fun_with_wget/