Re:Nordstrom's robots.txt
by IgorPartola on Friday February 27, 2015 @05:03PM
(#9121651)
Could that be a honeypot to trap bots that don't respect robots.txt?
[ Parent
| on HN ]
Re:Nordstrom's robots.txt
by huhtenberg on Friday February 27, 2015 @06:03PM
(#9121934)
Looks more like a mop-up after a website infection incident.
I have had to do that before. The URLs in that case seemed to have a more random distribution than most of these.
maybe, but why would THAT be the honey pot? it could be any phrase, really.
Re:Nordstrom's robots.txt
by IgorPartola on Friday February 27, 2015 @05:14PM
(#9121709)
Well, you'd want them to try to crawl it. Perhaps there was a bot crawling their site for such links and they wanted to identify specifically that one. It seems most likely that such URL's never existed, but this would be a decent way to identify a bot that's otherwise not distinguishable.