News for hackers. Stuff that matters.
Re:Robots.txt by fein on Monday February 11, 2013 @01:20PM (#5202262)
I would hope that google, of all companies, would be capable of writing a crawler that ignores robots.txt comments.

Other than that the guy only has one rule.

There are no comments on this comment yet.