Re:Nordstrom's robots.txt
by nottestuser on Friday February 27, 2015 @04:49PM
(#9121579)
You're rolling along thinking this is just some bad web developer somewhere until you get to the last Disallow...
[ Parent
| on HN ]
Re:Nordstrom's robots.txt
by IgorPartola on Friday February 27, 2015 @05:03PM
(#9121651)
Could that be a honeypot to trap bots that don't respect robots.txt?
Re:Nordstrom's robots.txt
by huhtenberg on Friday February 27, 2015 @06:03PM
(#9121934)
Looks more like a mop-up after a website infection incident.
I have had to do that before. The URLs in that case seemed to have a more random distribution than most of these.
maybe, but why would THAT be the honey pot? it could be any phrase, really.
Re:Nordstrom's robots.txt
by IgorPartola on Friday February 27, 2015 @05:14PM
(#9121709)
Well, you'd want them to try to crawl it. Perhaps there was a bot crawling their site for such links and they wanted to identify specifically that one. It seems most likely that such URL's never existed, but this would be a decent way to identify a bot that's otherwise not distinguishable.
Well, I'm on a list now..
You have something against petite young ladies of a modest stature? Sizeist!
Re:Nordstrom's robots.txt
by shouldbeworking on Friday February 27, 2015 @07:27PM
(#9122229)
This other retailer's page is about the same except for that one entry: http://www.neimanmarcus.com/robots.txt
I don't get it: is that disallow targeting some sort of porn spambot crawler?
They probably made a product with that slug by accident and are doing that to try to get it removed from search engine results.
It makes me wonder if Nordstrom ever stocked the now-defunct Pornstar clothing brand.
Re:Nordstrom's robots.txt
by Springtime on Friday February 27, 2015 @04:58PM
(#9121628)
Disallow is used to exclude certain URLs from being crawled, in this case it looks like a search query as the store's search uses this pattern: http://shop.nordstrom.com/sr/query .
Re:Nordstrom's robots.txt
by philip1209 on Friday February 27, 2015 @04:55PM
(#9121617)
Maybe some mal-intended inbound links that they don't want to be associated with?
Is this some sort of new terrible negative seo? Associate a target url for a bunch of negative things?