News for hackers. Stuff that matters.

A comment on Google's robots.txt. Parent comment

Re:Google's robots.txt by mkonecny on Saturday February 22, 2014 @11:08PM (#7284882)
It just issues a "HTTP/1.1 302 Moved Temporarily" directed to their homepage. Requesting an invalid file such as "robots.txtsdfa32r523" has the same effect, so they probably don't have a robots file at all.
[ Parent | on HN ]
Re:Google's robots.txt by blossoms on Saturday February 22, 2014 @11:59PM (#7284993)
http://i.imgur.com/Jmjf3y2.png
[ Parent | on HN ]
Re:Google's robots.txt by judk on Saturday February 22, 2014 @11:12PM (#7284894)
Huh? No, it is a regular robots.txt file
[ Parent | on HN ]
Re:Google's robots.txt by tonyedgecombe on Sunday February 23, 2014 @02:57AM (#7285286)
It redirects requests from the UK.
[ Parent | on HN ]