Yeah, I agree with you for the most part. And this argument comes up a lot. The difference is we're not Amazon yet. We don't have the same levels of user engagement, loyalty and whatever metrics that makes Amazon a success. To the end users, we're not as popular, one slow page load translates to a lost potential customer which when aggregated to even a few 100 potential customers could make or break a startup.
But let us be realistic. I just loaded www.amazon.com - 17.8 MB transferred when it finally stopped. Arguably one of the most successful websites in the world.
A 5MB asset size for most sites is perfectly acceptable.