News for hackers. Stuff that matters.
Re:Gone in Six Characters: Short URLs Considered Harmful for... by _Understated_ on Friday April 15, 2016 @11:22AM (#11505032)
Following the rules laid out in a robots.txt file is optional. The reputable search engines tend to play by the rules but dodgy ones? Not so sure.
Re:Gone in Six Characters: Short URLs Considered Harmful for... by ikeboy on Friday April 15, 2016 @11:44AM (#11505202)
The dodgy ones don't have access to the URLs. The example was Skype links: if Microsoft scrapes those, they'll follow the rules and not make them available in searches.

If you give your link to a dodgy search engine, you've lost.