News for hackers. Stuff that matters.
Re:Robots.txt is a suicide note by sbierwagen on Wednesday May 11, 2011 @02:59PM (#2537858)
Well, when I iterated through the everything2.com namespace and downloaded the majority of their content[1], I respected their robots.txt. After I did that, they changed it to:

  User-agent: * 
  Disallow: /
Does that count?

1: http://bbot.org/blog/archives/2011/01/17/more_fun_with_wget/

There are no comments on this comment yet.