News for hackers. Stuff that matters.

A comment on Nordstrom's robots.txt. Parent comment

Re:Nordstrom's robots.txt by troels on Friday February 27, 2015 @06:55PM (#9122133)
I don't think you actually want to do that. Blocking in robots.txt will prevent Google from crawling the URL - not from indexing it. You actually want them to crawl the URL and then respond with a 404 or 410.

If there are inbound links pointing to that URL, you should disavow them in GWT.

[ Parent | on HN ]
Re:Nordstrom's robots.txt by eli on Friday February 27, 2015 @07:20PM (#9122206)
a "noindex" robots meta tag would work too
[ Parent | on HN ]