Re:Shodan: The scariest search engine on the Internet
by pseut on Monday April 08, 2013 @01:07PM
(#5512964)
Presumably Shodan ignores robots.txt, right?
[ Parent
| on HN ]
Re:Shodan: The scariest search engine on the Internet
by vidarh on Monday April 08, 2013 @06:07PM
(#5514980)
A "bad" search engine should treat robots.txt pretty much in reverse: Anything disallowed should go to the top of the list of things to index.. There are sites out there that uses robots.txt rules to prevent Google from indexing things that should be password protected but isn't...
Re:Shodan: The scariest search engine on the Internet
by nostrademons on Tuesday April 09, 2013 @09:55PM
(#5522776)
The irony is that robots.txt doesn't even prevent things from being indexed. The files can still be indexed if there's a link to them on the Internet; that's what <meta noindex> is for. (Which, ironically, requires that the page not be robotted, because if it is it can't be crawled, which means the meta tag can't be discovered.)