News for hackers. Stuff that matters.
Re:Shodan: The scariest search engine on the Internet by nostrademons on Tuesday April 09, 2013 @09:55PM (#5522776)
The irony is that robots.txt doesn't even prevent things from being indexed. The files can still be indexed if there's a link to them on the Internet; that's what <meta noindex> is for. (Which, ironically, requires that the page not be robotted, because if it is it can't be crawled, which means the meta tag can't be discovered.)

There are no comments on this comment yet.