News for hackers. Stuff that matters.
Re:Nordstrom's robots.txt by nottestuser on Friday February 27, 2015 @04:49PM (#9121579)
You're rolling along thinking this is just some bad web developer somewhere until you get to the last Disallow...
Re:Nordstrom's robots.txt by IgorPartola on Friday February 27, 2015 @05:03PM (#9121651)
Could that be a honeypot to trap bots that don't respect robots.txt?
Re:Nordstrom's robots.txt by huhtenberg on Friday February 27, 2015 @06:03PM (#9121934)
Looks more like a mop-up after a website infection incident.
Re:Nordstrom's robots.txt by ashmud on Friday February 27, 2015 @08:27PM (#9122395)
I have had to do that before. The URLs in that case seemed to have a more random distribution than most of these.
Re:Nordstrom's robots.txt by prokizzle on Friday February 27, 2015 @05:09PM (#9121681)
maybe, but why would THAT be the honey pot? it could be any phrase, really.
Re:Nordstrom's robots.txt by IgorPartola on Friday February 27, 2015 @05:14PM (#9121709)
Well, you'd want them to try to crawl it. Perhaps there was a bot crawling their site for such links and they wanted to identify specifically that one. It seems most likely that such URL's never existed, but this would be a decent way to identify a bot that's otherwise not distinguishable.
Re:Nordstrom's robots.txt by lfowles on Friday February 27, 2015 @04:51PM (#9121587)
Well, I'm on a list now..
Re:Nordstrom's robots.txt by nly on Friday February 27, 2015 @04:53PM (#9121598)
You have something against petite young ladies of a modest stature? Sizeist!
Re:Nordstrom's robots.txt by shouldbeworking on Friday February 27, 2015 @07:27PM (#9122229)
This other retailer's page is about the same except for that one entry: http://www.neimanmarcus.com/robots.txt
Re:Nordstrom's robots.txt by CoreSet on Friday February 27, 2015 @04:51PM (#9121588)
I don't get it: is that disallow targeting some sort of porn spambot crawler?
Re:Nordstrom's robots.txt by jarin on Friday February 27, 2015 @04:55PM (#9121612)
They probably made a product with that slug by accident and are doing that to try to get it removed from search engine results.
Re:Nordstrom's robots.txt by meej on Friday February 27, 2015 @05:52PM (#9121891)
It makes me wonder if Nordstrom ever stocked the now-defunct Pornstar clothing brand.
Re:Nordstrom's robots.txt by Springtime on Friday February 27, 2015 @04:58PM (#9121628)
Disallow is used to exclude certain URLs from being crawled, in this case it looks like a search query as the store's search uses this pattern: http://shop.nordstrom.com/sr/query .
Re:Nordstrom's robots.txt by philip1209 on Friday February 27, 2015 @04:55PM (#9121617)
Maybe some mal-intended inbound links that they don't want to be associated with?
Re:Nordstrom's robots.txt by cmdrfred on Friday February 27, 2015 @06:56PM (#9122138)
Is this some sort of new terrible negative seo? Associate a target url for a bunch of negative things?
Re:Nordstrom's robots.txt by prokizzle on Friday February 27, 2015 @04:52PM (#9121592)
Maybe that was their intention, but that's not how it's done.