News for hackers. Stuff that matters.
Re:Modern Web Applications are Here by jonknee on Tuesday November 15, 2011 @08:22AM (#3238090)
For most purposes, search engines seeing you = Googlebot seeing you. I have never had a competitor past single digits in percentage of search engine referrals.
Re:Modern Web Applications are Here by bad_user on Tuesday November 15, 2011 @11:34AM (#3238802)
A crawler is not necessarily providing content to a classical search engine, as it can serve many purposes. Ultimately such crawlers may generate links to your website and while not bringing you too much referrals traffic, it may increase your ranking, which ultimately means more traffic from Google.

Also, if search engine referrals is all you're interested in, if Google drops you from their index, your website might as well not exist, right? As I said, be careful not to throw the baby with the bathwater. Google is not what made the web great, but a Google dependency might break it and website authors can be blamed for this.