Re:Nordstrom's robots.txt
by troels on Saturday February 28, 2015 @05:54PM
(#9125357)
Yeah, the clue is in "Server: AkamaiGHost". The site is clearly behind some sort of anti-crawl protection which decides it doesn't like me. I think it would make sense to make an exception for exactly that file though.
$ curl --dump-header - 'http://shop.nordstrom.com/robots.txt' HTTP/1.1 200 OK Cache-Control: private Content-Type: text/html; charset=utf-8 Server: Microsoft-IIS/7.5 X-Powered-By: ASP.NET Content-Length: 865 Date: Sat, 28 Feb 2015 07:04:36 GMT Connection: keep-alive