News for hackers. Stuff that matters.
Re:Google has also indexed thousands of publicly accessible ... by nostrademons on Saturday January 26, 2013 @04:07PM (#5121758)
The way to prevent a site from being indexed at all is through a <meta name="robots" content="noindex,nofollow"> tag on the page or X-Robots-Tag HTTP header (both of which, ironically, require that you not robots.txt it out, because otherwise the page content will never be crawled), or through a Noindex directive in robots.txt (which is unspecified by the spec - Google supports it, but Yahoo and Bing don't).

There are no comments on this comment yet.