Nordstrom's robots.txt

Re:Nordstrom's robots.txt by olivierkaisin on Friday February 27, 2015 @05:25PM (#9121763)

I wouldn't be surprised to hear that they did put that url into their robots.txt because it is a search result page generated by a bot and that has been visited and indexed by Google…

[ Parent | on HN ]

Re:Nordstrom's robots.txt by hurin on Friday February 27, 2015 @05:55PM (#9121904)

I'm not up to date on how google's current search indexing algorithm works. Supposing someone has a lot of bots that post links to `extra-small-teen-pony` or some other /sr query on nordstrom.com from other sites - would google index that?

[ Parent | on HN ]

Re:Nordstrom's robots.txt by cmdrfred on Friday February 27, 2015 @06:48PM (#9122107)

If they threw a custom 404 that reports 200. That's how I assumed those phone number lookup sites work.

[ Parent | on HN ]