However in working on the application web for the last 5 years I've come to appreciate the power of new approaches for web-based user applications. It turns out that generating full HTML pages for users can turn into an incredible waste of resources and significantly worse UX, for what amounts to a very tiny accessibility and SEO advantage. Facebook is the canonical example here. The way they load a page is brilliant from a UX and server resource perspective.
But consider apps that are much less dynamic. Say you have a 50k page with 1k of dynamic content per user or some other variable such as country. Page caching is impossible. You can rig up all kinds of complex server side fragment caching mechanisms, and squeeze quite a bit of performance out. However these schemes quickly grow very complex, whereas the client/server architecture lends itself to a solution that is orders of magnitude more elegant, which is simply to load the dynamic bits via AJAX. This A) offloads processing to the client, B) solves all kinds of caching problems, allowing full HTTP caching where it would have been impossible, and bypasses hordes of thorny cache invalidation problems, C) creates some elegant optimizations that would have chicken-or-egg problems if attempted on a server-side MVC architecture alone, D) Improves page responsiveness dramatically as the initial HTML shell and static assets can start loading almost instantaneously, and E) allows a more de-coupled architecture that makes sharding and scaling much easier.
Because of the technical advantages, it's inevitable that web-based app development will continue moving in this direction. Web apps that don't go in this direction will be at a disadvantage from an operating cost and UX perspective.
http://code.google.com/web/ajaxcrawling/docs/getting-started...
that's where that exclamation mark in the hash of the URL in recent sites comes from (like new twitter)
Edit:
Look at this query:
http://www.google.com/search?sourceid=chrome&ie=UTF-8...
it finds the existence of that word on my privacy page that gets only loaded via AJAX. So it must be able execute the javascript (sammy, basically) that intercepts these links to load the correct page fragment.
Edit:
Explanation is wrong. It takes a little bit of server side magic to work. See comment below. I completely forgot about this after implementing it in April :-)
http://tempalias.com/#!/privacy
Gets converted by their crawler in to this:
http://tempalias.com/?_escaped_fragment_=/privacy
It's up to you to implement your server-side code such that ?_escaped_fragment_ causes an indexable version of your content to be served up.
However, something like login management could be completely handled by an API. So the browser would be combining different sources of HTML into once package. This is pretty much what is happening with the Facebook "Like" button anyway.
There are a lot of sites out there which serve pages which are mostly static content with just a little bit extra -- the login name at the top, personalized tools here and there.
I happen to work with Wikipedia content, so that's top of my mind right now. If you are logged in, we give you content rendered mostly from scratch, since the caching system is optimized for the non-logged-in user and there are user preferences which might transform the content in complicated ways. (Even things like date formats are subject to user preferences, so it's considerably harder to cache). This works okay, but it's dramatically more expensive for logged in users. We would have to rethink things if we ever got to a world where a lot more people are logged in.
A paradigm I'd like to explore would be to fetch the static content from one set of servers, and login state and personal options and other transformations from another, and get the browser to combine it all. Probably won't ever happen on Wikipedia though, at least not soon.