News for hackers. Stuff that matters.
Re:What Google Won't Let MS Click by gojomo on Wednesday September 12, 2007 @07:58PM (#54070)
Misleading headline: the disallows affect all crawlers equally.

Missing context: what is someone, expert at reading robots.txt or not, expected to find interesting in the target link?

Bad submission.

Re:What Google Won't Let MS Click by Goladus on Wednesday September 12, 2007 @10:08PM (#54114)
robots.txt is not a security feature. It's a courtesy that saves resources and makes life a lot easier for everyone who is willing to play by the rules.
Re:What Google Won't Let MS Click by rainsill on Thursday September 13, 2007 @06:40PM (#54425)
This is a bad submission. False headline. How do you vote an article down?
Re:What Google Won't Let MS Click by rms on Thursday September 13, 2007 @07:18PM (#54435)
Making a comment saying it is a bad submission is great, just please don't say "This article should be on reddit!!!!!!!!"
Re:What Google Won't Let MS Click by chengmi on Wednesday September 12, 2007 @06:33PM (#54049)
Does this even matter? How many bots actually respect robots.txt?