That's a big part of what people don't realize - city centers and old neighborhoods will have houses of vastly different ages (my town's center has houses that were built this year next to houses that are almost 200 years old).
But most of the houses in America aren't those. They're in developments of different ages, and the houses in those developments look roughly the same. You might notice some differences in the very old 40s developments, because of 80 years of additions and remodeling, but most of the houses will have survived and few will have been rebuilt.
When a city builds up slowly, you get many different things mixed together, and that is more interesting.
Clearly you and I disagree, for instance, on if victorian houses can be aesthetically pleasing on their own, but would you rather have a street full of plywood boxes or would you rather have my victorian-styled house sitting next to your [whatever] and someone else's [whatever]? I'd find that great, and much more interesting than a street full of near-identical ones.
(But in practice - even without "cultural unity" - that isn't what most new development in the US produces today - rather, you have a block-or-larger development of a bunch of near-identical [something]-esque generic builds. Unity achieved! Aesthetic appeal? Maybe not so much...)