Re:Shodan: The scariest search engine on the Internet
by nostrademons on Tuesday April 09, 2013 @09:55PM
(#5522776)
The irony is that robots.txt doesn't even prevent things from being indexed. The files can still be indexed if there's a link to them on the Internet; that's what <meta noindex> is for. (Which, ironically, requires that the page not be robotted, because if it is it can't be crawled, which means the meta tag can't be discovered.)
[ Parent
| on HN ]
There are no comments on this comment yet.