Nordstrom's robots.txt

Re:Nordstrom's robots.txt by WestCoastJustin on Friday February 27, 2015 @05:08PM (#9121676)

Reading, reading, reading... wtf! I hope that is an abbreviation for something, but to be honest, I don't want to Google and find out.

  Disallow: /sr/extra-small-teen-porn*

[ Parent | on HN ]

Re:Nordstrom's robots.txt by untog on Friday February 27, 2015 @05:16PM (#9121722)

I'm not sure what letters would go on the end of "porn" to make a clothing item, but I'll bet it exists. Or someone misspelt "pony", or... well, you get the idea. No matter what, you really don't want your site to appear in search results for those terms.

[ Parent | on HN ]

Re:Nordstrom's robots.txt by omarish on Friday February 27, 2015 @06:02PM (#9121933)

➜ ~ cat /usr/share/dict/words | grep "^porn"

pornerastic

pornocracy

pornocrat

pornograph

pornographer

pornographic

pornographically

pornographist

pornography

pornological

[ Parent | on HN ]

Re:Nordstrom's robots.txt by jjoonathan on Friday February 27, 2015 @06:38PM (#9122072)

> pornocracy

Is this what we will call it when the NSA starts blackmailing politicians with their internet histories?

[ Parent | on HN ]

Re:Nordstrom's robots.txt by 1ris on Friday February 27, 2015 @08:02PM (#9122335)

It means rule of the whores and is a certain period in the catholic church.

[ Parent | on HN ]

Re:Nordstrom's robots.txt by DanBC on Friday February 27, 2015 @08:04PM (#9122342)

http://en.wikipedia.org/wiki/Saeculum_obscurum

> Saeculum obscurum (Latin: the Dark Age) is a name given to a period in the history of the Papacy during the first half of the 10th century, beginning with the installation of Pope Sergius III in 904 and lasting for sixty years until the death of Pope John XII in 964. During this period, the Popes were influenced strongly by a powerful and corrupt aristocratic family, the Theophylacti, and their relatives.

[ Parent | on HN ]

Re:Nordstrom's robots.txt by cbd1984 on Friday February 27, 2015 @07:32PM (#9122245)

Why are you catting a single file into grep?

[ Parent | on HN ]

Re:Nordstrom's robots.txt by WestCoastJustin on Friday February 27, 2015 @07:46PM (#9122279)

Muscle memory. I find myself doing this all the time too, it is just a universal way of filtering things down, so I don't worry about it too much. Like when using lsof, sure you can use filters via lsof itself, but sometimes you just use "lsof |grep" because it is easier than looking up the correct option. The performance to muscle memory tradeoff is worth it.

[ Parent | on HN ]

Re:Nordstrom's robots.txt by juliangregorian on Saturday February 28, 2015 @01:19AM (#9122967)

I think GP may be referring to the "unnecessary use of cat"

[ Parent | on HN ]

Re:Nordstrom's robots.txt by cbd1984 on Friday February 27, 2015 @08:54PM (#9122465)

The correct way:

grep "^porn" /usr/share/dict/words

[ Parent | on HN ]

Re:Nordstrom's robots.txt by michaelmior on Saturday February 28, 2015 @12:48AM (#9122921)

Nothing incorrect about using cat. It makes almost zero practical difference except saving a couple keystrokes. There are some scenarios where it actually matters, but this isn't one of them.

[ Parent | on HN ]

Re:Nordstrom's robots.txt by woonord on Friday February 27, 2015 @05:26PM (#9121764)

impossible to do more research on this at work.

[ Parent | on HN ]

Re:Nordstrom's robots.txt by patcon on Friday February 27, 2015 @11:32PM (#9122790)

shop.nordstrom.com/sr is for search results. So I guess someone linked to that search result somewhere, likely as a joke

EDIT: Looks like "porn star clothing" search also got indexed: https://www.google.ca/search?client=ubuntu&channel=fs&q=porn... (apparently someone has an edgy blog where that link is the punchline?)

[ Parent | on HN ]

Re:Nordstrom's robots.txt by pvsnp on Friday February 27, 2015 @05:11PM (#9121693)

That's just way too specific to be an accident, wonder what the story is..

[ Parent | on HN ]

Re:Nordstrom's robots.txt by olivierkaisin on Friday February 27, 2015 @05:25PM (#9121763)

I wouldn't be surprised to hear that they did put that url into their robots.txt because it is a search result page generated by a bot and that has been visited and indexed by Google…

[ Parent | on HN ]

Re:Nordstrom's robots.txt by hurin on Friday February 27, 2015 @05:55PM (#9121904)

I'm not up to date on how google's current search indexing algorithm works. Supposing someone has a lot of bots that post links to `extra-small-teen-pony` or some other /sr query on nordstrom.com from other sites - would google index that?

[ Parent | on HN ]

Re:Nordstrom's robots.txt by cmdrfred on Friday February 27, 2015 @06:48PM (#9122107)

If they threw a custom 404 that reports 200. That's how I assumed those phone number lookup sites work.

[ Parent | on HN ]