It depends™ on your usecases. Personally, I stream YouTube at 360p most of the time because the extra pixels just don't matter (this is for entertainment; screencasts and the like need more for legible text). Most of my downloads run async (OS updates, packages to install, etc). I'm on a shared connection but we don't seem to have issues.

Of course, you're right; all other things held fixed I'd rather have a faster connection. But in practice the cost is such that I'd want a compelling reason, and I just don't care that much.