News for hackers. Stuff that matters.
Re:Nordstrom's robots.txt by troels on Friday February 27, 2015 @06:46PM (#9122100)
Hmm ...

    $ curl --dump-header - 'http://shop.nordstrom.com/robots.txt'
    HTTP/1.1 403 Forbidden
    Server: AkamaiGHost
    Mime-Version: 1.0
    Content-Type: text/html
    Content-Length: 281
    Expires: Fri, 27 Feb 2015 23:42:13 GMT
    Date: Fri, 27 Feb 2015 23:42:13 GMT
    Connection: close
    ...
Probably a bad idea to deny serving robots.txt to bots (even if they ought to interpret that as a total ban)
Re:Nordstrom's robots.txt by felixvolny on Saturday February 28, 2015 @02:06AM (#9123037)
Funny, when I do the same thing I get a completely different response:

$ curl --dump-header - 'http://shop.nordstrom.com/robots.txt' HTTP/1.1 200 OK Cache-Control: private Content-Type: text/html; charset=utf-8 Server: Microsoft-IIS/7.5 X-Powered-By: ASP.NET Content-Length: 865 Date: Sat, 28 Feb 2015 07:04:36 GMT Connection: keep-alive

Re:Nordstrom's robots.txt by troels on Saturday February 28, 2015 @05:54PM (#9125357)
Yeah, the clue is in "Server: AkamaiGHost". The site is clearly behind some sort of anti-crawl protection which decides it doesn't like me. I think it would make sense to make an exception for exactly that file though.