Google's robots.txt

Re:Google's robots.txt by blossoms on Saturday February 22, 2014 @11:06PM (#7284876)

Unrelated but it looks like www.aol.com's robots.txt is served as text/html http://www.aol.com/robots.txt

Is this a common mistake?

[ Parent | on HN ]

Re:Google's robots.txt by saltysugar on Sunday February 23, 2014 @02:03AM (#7285194)

Yes, from the server side's mishandling of TXT extension. Probably the server put the MIME type in the HTTP header as "HTML" instead of TXT, and the browser renders the page as such.

[ Parent | on HN ]

Re:Google's robots.txt by pjscott on Sunday February 23, 2014 @02:27AM (#7285238)

From the headers:

    Content-Type: text/html;charset=UTF-8

It also tries to set no less than four cookies.

[ Parent | on HN ]

Re:Google's robots.txt by mkonecny on Saturday February 22, 2014 @11:08PM (#7284882)

It just issues a "HTTP/1.1 302 Moved Temporarily" directed to their homepage. Requesting an invalid file such as "robots.txtsdfa32r523" has the same effect, so they probably don't have a robots file at all.

[ Parent | on HN ]

Re:Google's robots.txt by blossoms on Saturday February 22, 2014 @11:59PM (#7284993)

http://i.imgur.com/Jmjf3y2.png

[ Parent | on HN ]

Re:Google's robots.txt by judk on Saturday February 22, 2014 @11:12PM (#7284894)

Huh? No, it is a regular robots.txt file

[ Parent | on HN ]

Re:Google's robots.txt by tonyedgecombe on Sunday February 23, 2014 @02:57AM (#7285286)

It redirects requests from the UK.

[ Parent | on HN ]