Re:Ask HN: Is security by obfuscation sufficient?
by andrewljohnson on Monday November 16, 2009 @05:28PM
(#945462)
It's also worth pointing out that only the likes of Google and Yahoo will even look at robots.txt.
Some other crawler may walk in there, then create pages that link your files, and then Google could link those pages, and it's game over.
But if you put exact URLs in your robots.txt, then a normal person could pick them up from there and access your content.
Make sure you use a directory structure and forbid all files from the content directory :-)