News for hackers. Stuff that matters.
Re:Modern Web Applications are Here by bad_user on Tuesday November 15, 2011 @07:32AM (#3238000)
Only Google does that currently.

And while people can build crawlers that can fetch Javascript rendered pages, these scripts will have to drive real browsers with rendering engines, not just simple HTTP clients. And this is a problem, because fetching millions of web pages will require serious infrastructure work, versus a crawler using libev that can fetch dozens of pages in parallel without a sweat.

Also, that spec you gave is not exactly how Google operates. That's the advice Google gives for making web apps compliant, but the reality is that Google is smarter than that and it takes Google a good amount of effort to fix other people's screwups.

Accessible to crawlers doesn't mean Google-only and you should be careful not to throw the baby along with the bathwater.

Re:Modern Web Applications are Here by jonknee on Tuesday November 15, 2011 @08:22AM (#3238090)
For most purposes, search engines seeing you = Googlebot seeing you. I have never had a competitor past single digits in percentage of search engine referrals.
Re:Modern Web Applications are Here by bad_user on Tuesday November 15, 2011 @11:34AM (#3238802)
A crawler is not necessarily providing content to a classical search engine, as it can serve many purposes. Ultimately such crawlers may generate links to your website and while not bringing you too much referrals traffic, it may increase your ranking, which ultimately means more traffic from Google.

Also, if search engine referrals is all you're interested in, if Google drops you from their index, your website might as well not exist, right? As I said, be careful not to throw the baby with the bathwater. Google is not what made the web great, but a Google dependency might break it and website authors can be blamed for this.