News for hackers. Stuff that matters.
Re:Making AJAX applications crawlable by pseudonym on Friday October 01, 2010 @10:32AM (#1747120)
Considering that it's Google, I'm honestly surprised they haven't created a crawler application that can execute the javascript on a given page and re-parse it based on the new layout.

You wouldn't think it would be that hard, either-- take Chrome, remove UI, add crawler. Chrome's already got the functionality in "Inspect Element" to show dynamically-created content.

Re:Making AJAX applications crawlable by patio11 on Friday October 01, 2010 @11:43AM (#1747373)
Google does both heuristically parse Javascript and execute Javascript, for at least some fraction of their crawl. This is obviously much more expensive than doing HTML parsing, particularly once the Internet largely goes from static HTML to AJAXy magic.
Re:Making AJAX applications crawlable by pseudonym on Friday October 01, 2010 @12:17PM (#1747511)
Granted, but if Google can't parse it, I highly doubt anyone else is going to. And as nice as it would be if every web developer read the "Google Guide to Being Nice to Our Web Crawlers", as an internet we still can't get away from IE6. The term "pipe dream" comes to mind.
Re:Making AJAX applications crawlable by eli on Friday October 01, 2010 @11:19AM (#1747293)
Re:Making AJAX applications crawlable by paradoja on Friday October 01, 2010 @10:41AM (#1747161)
That would be a fairly bad idea. Javascript execution, and in particular AJAX calls, frequently change data. I wouldn't want my crawler to delete or edit somehow lots of data if I where Google.
Re:Making AJAX applications crawlable by pseudonym on Friday October 01, 2010 @10:55AM (#1747208)
No more or less than following a link with a GET variable. I'm trying to find the article posted on TDWTF regarding a link that went to "&delete=true" that dropped the entire database...but, Ajax calls are no more or less frequently data-changing than any one of a hundred links on survey sites that send their data with GET variables.

Edit: Found it! http://thedailywtf.com/Articles/WellIntentioned-Destruction....

Re:Making AJAX applications crawlable by RyanMcGreal on Friday October 01, 2010 @11:29AM (#1747323)
As long as the crawler can't execute POST requests, no one who has built their web application properly will have any problems.
Re:Making AJAX applications crawlable by njharman on Friday October 01, 2010 @12:27PM (#1747544)
> no one who has built their web application properly will have any problems

In other words everyone will have problems (to an accuracy of 3-4 decimals)

Re:Making AJAX applications crawlable by RyanMcGreal on Friday October 01, 2010 @12:39PM (#1747602)
Re:Making AJAX applications crawlable by IgorPartola on Friday October 01, 2010 @01:45PM (#1747846)
And how would the crawler know when the website is done loading?
Re:Making AJAX applications crawlable by wanderr on Saturday October 02, 2010 @11:29PM (#1752013)
I wish they would make it possible to do the whole bit with "pretty" urls. Example:

Users go to: /#/abc Google goes to /abc and they both see the same content. Obviously you'd need some way to tell Google that your site follows this scheme but then you don't have to change all of your URLs and existing external links to pages on your site would accumulate page rank appropriately.

Re:Making AJAX applications crawlable by timinman on Friday October 01, 2010 @05:46PM (#1748900)
I've been thinking about writing a blog article on this subject. Basically AJAX is like Chocolate - it's really tempting, but eating too much or for the wrong reasons is not a good idea.

I've just rewritten one of my sites which used AJAX to load content on a single index page. I've now made sure every unique resource has a unique url. This is good for browser navigation, good for Google, and good for promoting a single resource on a service like Twitter. Now my site which looked like 1 page to Google is almost 500 pages.

Re:Making AJAX applications crawlable by voxxit on Friday October 01, 2010 @08:26PM (#1749304)
I just recently implemented this. The thing I'm having trouble with is the fact that they don't limit the amount of time (or specify a limit, rather) which is acceptable to make the crawler wait. For instance, what if it take 20 seconds to load an acceptable amount of JavaScript-created HTML?
Re:Making AJAX applications crawlable by ajennings on Friday October 01, 2010 @03:07PM (#1748190)
Does anyone know when this went live? I'm assuming it is live already.

I see it was proposed in Oct 2009: http://googlewebmastercentral.blogspot.com/2009/10/proposal-...