AAAAAA
CBBBBB
CBBBBB
CBBBBB
CDDDDD
CDDDDD
CDDDDD
CDDDDD
Most users will have their screen break a bit above the end of the B block. (By design.)
The heaviest part of the page is D. Eliminating it is not an attractive option -- the overall conversion rate suffers. The most valuable parts of the conversion pathway are in B and C. The first thing the customer sees is invariably A.
Solution: a bit of technical magic to make sure that the website loads element in A->C->B->D order (there is a competing technical imperative which means that in the HTML it makes more sense to order them A->B->D->C), and significant microoptimization of how quickly B and C are minimally interactive to the user. (We're talking measuring improvements in the kilobytes or tens of microseconds range.) Since customers have to actively scroll down to engage the D block, and since most do not engage the D block before first engaging the B block, the fact that D is heavy and slow very rarely impacts their experience.
That link is a talk on YSlow, a tool you can use to lower download speed, render time, and perceived load time. It's awesome.
That article also links to a great article called "Maximizing Human Performance" (http://www.asktog.com/basics/03Performance.html) which goes over a few topics, including response times and their perceived meaning and optimizing user flows for the human thinking process.
We used to have a webform that (in summary) asked the user what they wanted to download, how they got there, and how we can contact them later (as sales leads). Conversion ratio was fairly poor, so I refactored the priorities of what we were asking for into 2 forms.
The first form consisted of questions we HAD to know the answer to to service the user's request, and the second form consisted of info we wanted from them (name, email, contact info, etc.)
So between the first form and the second form, we'd kick off the download for the user (and show a progress meter at the top) -- which got them what they needed immediately, making the site seem faster overall.
Plus, since they were waiting for the download to complete anyway, the conversion goals for the information we wanted went way, way up as well.
Again, I didn't actually MAKE things faster, in fact, having to fill out 2 forms was significantly slower, but we kept users engaged and let them waste their idle cycles making us happy since we were already accomodating their request.
Known tricks are explicit sizes for tables and images. The HTML file itself is loaded fast so you can start reading while images are loading.
There's also the opposite method: preloading images so, once the needed items are in the browser, the page renders faster.
I wonder how it would work to preload the images for the initially visible part of a page and leave for later the ones that you must scroll to see.
I've read recently about using sprites for the graphics: one big image with a mosaic of little images used, then split using CSS. This trick reduces the number of HTTP requests.
Yahoo has an open source js library that does exactly that: http://developer.yahoo.com/yui/imageloader/
You could separate it into 2 segments. One portion of users not having used a website in a particular category and the other bucket being folks that had used such websites before. Let's take online travel websites as an example.
In both cases people had a good experience using the website which in turn was a combination of great design including crispness & information architecture, variable font sizes, great presentation of the most important information, a lot of user empathy, generally not making the user think and not surprising the user, heat maps of which areas of a webpage users are likely to look at, etc.
The technical aspects of it included, smaller webpage sizes, tons of caching by abstracting away personalized portions of the webpage, and thereby caching close to 70-95% of the webpage in the browser, using a CDN, compression, reducing RTT, using distributed dns, not accessing the database for requests if possible, having as many static components as possible, use of persistent connections, caching url query strings, and all the things mentioned in Y! performance guidelines.
For folks that had never used an online travel website, they compared the speed of a website to the ones they regularly used like google, rediff, yahoo, etc. For folks that had used an online travel website, the comparison was with the competition and how much better you were with the above aspects.