News for hackers. Stuff that matters.

A comment on Robots.txt is a suicide note. Parent comment

Re:Robots.txt is a suicide note by chalst on Wednesday May 11, 2011 @01:54AM (#2535328)
Do all of them ignore robots.txt?
[ Parent | on HN ]
Re:Robots.txt is a suicide note by sbierwagen on Wednesday May 11, 2011 @02:59PM (#2537858)
Well, when I iterated through the everything2.com namespace and downloaded the majority of their content[1], I respected their robots.txt. After I did that, they changed it to:

  User-agent: * 
  Disallow: /
Does that count?

1: http://bbot.org/blog/archives/2011/01/17/more_fun_with_wget/

[ Parent | on HN ]