News for hackers. Stuff that matters.
Re:Shodan: The scariest search engine on the Internet by pseut on Monday April 08, 2013 @01:07PM (#5512964)
Presumably Shodan ignores robots.txt, right?
Re:Shodan: The scariest search engine on the Internet by vidarh on Monday April 08, 2013 @06:07PM (#5514980)
A "bad" search engine should treat robots.txt pretty much in reverse: Anything disallowed should go to the top of the list of things to index.. There are sites out there that uses robots.txt rules to prevent Google from indexing things that should be password protected but isn't...
Re:Shodan: The scariest search engine on the Internet by nostrademons on Tuesday April 09, 2013 @09:55PM (#5522776)
The irony is that robots.txt doesn't even prevent things from being indexed. The files can still be indexed if there's a link to them on the Internet; that's what <meta noindex> is for. (Which, ironically, requires that the page not be robotted, because if it is it can't be crawled, which means the meta tag can't be discovered.)
Re:Shodan: The scariest search engine on the Internet by achillean on Monday April 08, 2013 @06:09PM (#5514992)
I don't crawl for URLs so it doesn't really factor into the equation for me.