This is true, and the U.S. road network evolved this way, with cars evolving to suit the 'freeway' or the little road to get to the 'freeway'.
In Europe we have an entirely different type of road network, ours does have fast motorways and dual carriageways where 'handling is not needed'. However, the bulk of the network is the older type of network where this handling thing does matter. In fact, we kind of chuckle with embarrassment at the sight of U.S. cars and their crude engineering. In a narrow country lane when the 325i meets the U.S. soccer-mom lard-mobile, you know that the guy in the 325i will be the one flicking into reverse and backing up to a passing point.
At worst, "handling" is the difference between a car that's fun to drive and one that bores me. At best, it's the difference between smashing into the truck that stops suddenly at highway speeds and making it onto the shoulder.
I drive a small family car and I know that with worse handling we'd all have been to the hospital at least twice.
If your experience with driving is sitting in traffic or driving around suburban streets, you're right, handling (mostly) doesn't matter.
[1] https://s-media-cache-ak0.pinimg.com/originals/ca/e2/45/cae2...
But I admit I might feel differently if I had to navigate roads like that on a daily basis.
I've been a driver for twenty years and I have never felt like I cared about 'handling'. When car reviewers talk about it I always picture a 45 year old man-child white-knuckling the wheel at the local intersection while revving his 325i, eyes darting between soccer moms and pensioners in the cars around him and imagining himself to be Ayrton Senna. Though that is probably uncharitable :)
I have owned many different cars between a top-end BMW and a beater Hyundai. The BMW is certainly a more enjoyable ride, but 'handling' was never a concern, except perhaps in a very occasional circumstance that might have gotten me arrested.