News for hackers. Stuff that matters.
Re:The Definition of Evil: Microsoft's Search Wars Hurt Us All by itgoon on Monday November 23, 2009 @04:16PM (#958163)
I think not heeding robots.txt would cause them more problems than it would solve.

Currently, it gets used for more than "deny/allow all" - it is selective. If I have to worry about stuff I don't want indexed being indexed, then I'm going to block all crawling. Yes, I know that wouldn't be easy, but I wouldn't be alone in my sentiment.

There are no comments on this comment yet.