Re:Nordstrom's robots.txt
by CoreSet on Friday February 27, 2015 @04:51PM
(#9121588)
I don't get it: is that disallow targeting some sort of porn spambot crawler?
[ Parent
| on HN ]
They probably made a product with that slug by accident and are doing that to try to get it removed from search engine results.
It makes me wonder if Nordstrom ever stocked the now-defunct Pornstar clothing brand.
Re:Nordstrom's robots.txt
by Springtime on Friday February 27, 2015 @04:58PM
(#9121628)
Disallow is used to exclude certain URLs from being crawled, in this case it looks like a search query as the store's search uses this pattern: http://shop.nordstrom.com/sr/query .
Re:Nordstrom's robots.txt
by philip1209 on Friday February 27, 2015 @04:55PM
(#9121617)
Maybe some mal-intended inbound links that they don't want to be associated with?
Is this some sort of new terrible negative seo? Associate a target url for a bunch of negative things?