News for hackers. Stuff that matters.
Re:Google has also indexed thousands of publicly accessible ... by gizmo686 on Friday January 25, 2013 @09:52PM (#5119023)
As franze pointed out, you can specify not to index in robots.txt (I have not confirmed this). The intent of dissalowing crawling is ambigous. Maybe they do not want their content cached, or the extra load on their server, or any number of reasons. If you need to de-index a site, you should use the robots.txt directive. If it has already been indexed and you need it de-indexed quickly, google offers tools to do so [1]

[1] http://support.google.com/webmasters/bin/answer.py?hl=en&...

Re:Google has also indexed thousands of publicly accessible ... by coderdude on Friday January 25, 2013 @10:06PM (#5119056)
Thank you for pointing that out to me.