I get that you’re basically right in technical terms. A nice condo development could meet all their physical needs in far less space. And we would all be better off as a society if that is what happened, if it could just spontaneously, voluntarily happen.

However it’s just not compatible with any concept of freedom, as understood by those people themselves. A lot of people have an expectation of fairness that they ought to be able to own a home of their choosing and stay in it until they’re ready to move (or dead). And they are sentimentally attached to their homes where they raised their children.

We in the West have a society whose framework is roughly aligned with that idea. We could change it, but it would require lots of other “the government knows best” policies, and ultimately the massive empowerment of police to enforce those “best practices” when people object. Even one would agree with the outcomes in housing policy, it would be pretty risky to create that kind of totalitarian situation in general, because we might not like the other “most efficient best practices.”