Re:Nordstrom's robots.txt
by troels on Saturday February 28, 2015 @05:54PM
(#9125357)
Yeah, the clue is in "Server: AkamaiGHost". The site is clearly behind some sort of anti-crawl protection which decides it doesn't like me. I think it would make sense to make an exception for exactly that file though.
[ Parent
| on HN ]
There are no comments on this comment yet.