Google has also indexed thousands of publicly accessible Panasonic webcams

Re:Google has also indexed thousands of publicly accessible ... by MiguelHudnandez on Friday January 25, 2013 @04:59PM (#5117924)

Ohh, got it, thank you. So Google is aware that the URL exists, even though they know nothing about the content served at that URL.

I am just surprised that a URL with no associated content would be included in the index.

But now that I think about it more, why not? It will not show up except in extremely specific searches, and in those cases it is useful to the searcher.

[ Parent | on HN ]

Re:Google has also indexed thousands of publicly accessible ... by coderdude on Friday January 25, 2013 @07:40PM (#5118649)

I find this behavior annoying. Here's why:

https://www.google.com/search?q=unicorn+admin

4th result down (wbpreview.com) is shown in search results despite blocking crawling/indexing with robots.txt. The result displays "A description for this result is not available because of this site's robots.txt – learn more" and the title seems to be auto-generated. The goal was to de-index the listing but apparently that's not an option.

[ Parent | on HN ]

Re:Google has also indexed thousands of publicly accessible ... by gizmo686 on Friday January 25, 2013 @09:52PM (#5119023)

As franze pointed out, you can specify not to index in robots.txt (I have not confirmed this). The intent of dissalowing crawling is ambigous. Maybe they do not want their content cached, or the extra load on their server, or any number of reasons. If you need to de-index a site, you should use the robots.txt directive. If it has already been indexed and you need it de-indexed quickly, google offers tools to do so [1]

[1] http://support.google.com/webmasters/bin/answer.py?hl=en&...

[ Parent | on HN ]

Re:Google has also indexed thousands of publicly accessible ... by coderdude on Friday January 25, 2013 @10:06PM (#5119056)

Thank you for pointing that out to me.

[ Parent | on HN ]

Re:Google has also indexed thousands of publicly accessible ... by nostrademons on Saturday January 26, 2013 @04:07PM (#5121758)

The way to prevent a site from being indexed at all is through a <meta name="robots" content="noindex,nofollow"> tag on the page or X-Robots-Tag HTTP header (both of which, ironically, require that you not robots.txt it out, because otherwise the page content will never be crawled), or through a Noindex directive in robots.txt (which is unspecified by the spec - Google supports it, but Yahoo and Bing don't).

[ Parent | on HN ]