Re:Ask HN: Has anyone ever gotten funding from this website?
by patio11 on Friday January 30, 2009 @12:33AM
(#457742)
spelled out like this so Google won't index it
Obscurity will NOT HELP YOU avoiding a Google index. They have many sources of data, including receiving toolbar data from users, and they very pointedly do not mention all sources they use to generate the crawl lists.
If you want to not get indexed, use the nonindex meta tag, or sign up for their webmaster console and remove that particular URL from the index. (Somewhat counterintuitively, robots.txt-ing out a site doesn't prevent it from being indexed, only from being crawled. They will still include it in their search results if it has external indicia of trust.)
Obscurity will NOT HELP YOU avoiding a Google index. They have many sources of data, including receiving toolbar data from users, and they very pointedly do not mention all sources they use to generate the crawl lists.
If you want to not get indexed, use the nonindex meta tag, or sign up for their webmaster console and remove that particular URL from the index. (Somewhat counterintuitively, robots.txt-ing out a site doesn't prevent it from being indexed, only from being crawled. They will still include it in their search results if it has external indicia of trust.)