News for hackers. Stuff that matters.

A comment on Nordstrom's robots.txt.

Re:Nordstrom's robots.txt by woonord on Friday February 27, 2015 @05:16PM (#9121720)
Hi! The robots file is not a text file. I know! Products blocked due to possible copyright infringement. A lot are disputes over product name or t-shirt slogan. We were ranking for some terms we did not want to be associated with so we added the offending search result page to the file. Not sure how it ever got indexed in the first place. Peace!
[ Parent | on HN ]
Re:Nordstrom's robots.txt by troels on Friday February 27, 2015 @06:55PM (#9122133)
I don't think you actually want to do that. Blocking in robots.txt will prevent Google from crawling the URL - not from indexing it. You actually want them to crawl the URL and then respond with a 404 or 410.

If there are inbound links pointing to that URL, you should disavow them in GWT.

[ Parent | on HN ]
Re:Nordstrom's robots.txt by eli on Friday February 27, 2015 @07:20PM (#9122206)
a "noindex" robots meta tag would work too
[ Parent | on HN ]
Re:Nordstrom's robots.txt by lucasjans on Saturday February 28, 2015 @12:04AM (#9122848)
Why you're at it - can you unblock traffic from Vietnam? I know a few people here who want to order online (through a courier) but can't and aren't capable of VPNing. Plus, isn't blocking entire countries from accessing your site against the spirit of the internet?

Edit: this is what we see from Vietnam - https://cloudup.com/cOLr9vimmIC

[ Parent | on HN ]
Re:Nordstrom's robots.txt by simonebrunozzi on Friday February 27, 2015 @05:19PM (#9121733)
I like you already :)
[ Parent | on HN ]
Re:Nordstrom's robots.txt by woonord on Friday February 27, 2015 @05:22PM (#9121749)
thanks! I have some inside knowledge!
[ Parent | on HN ]
Re:Nordstrom's robots.txt by toephu2 on Friday February 27, 2015 @05:58PM (#9121922)
as in you work at nordstrom?
[ Parent | on HN ]
Re:Nordstrom's robots.txt by woonord on Friday February 27, 2015 @06:06PM (#9121947)
kinda sorta maybe
[ Parent | on HN ]
Re:Nordstrom's robots.txt by filmgirlcw on Friday February 27, 2015 @05:21PM (#9121741)
awesome response!
[ Parent | on HN ]