The concept of low-rise residential & commercial is very recent, and was a reaction to the cold war (eg: lets spread out, nukes can't kill us all then!). You end up with sprawling burbs covering your best farmland, and a crapton of infrastructure to maintain at great expense. If it weren't for federal infrastructure subsidies covering the up front capital for infrastructure, most suburbs would not exist.
You can build dense, multi-story complexes with good insulation and seperation, were it not for running into your neighbors at the mailbox, you'd hardly see them at all. Most buildings built from 2002 on in Seattle are of this caliber, and having put up walls before, stapling some insulation between isn't that hard or expensive, and is worth every dollar if you plan to live there.
Getting into or out of a city is this recurring nightmare, of feeling trapped in a car or cramming into a train compartment with 250 other people.
We know how to fund, develop and create residential neighborhoods, but the interconnect with commercial and industrial zones, as lives change across decades becomes the challenge. People resist being packed together in stifling ways, and rightfully so. Some people hate the claustrophobia of cities, and some people chafe at simply having neighbors. Very few people would advocate for randomly assigned neighbors, if random includes ex-convicts for violent crimes, which becomes a reality in large cities.
If it were as simple as just building many reasonable houses, and bringing any people into them, to meet the needs of anyone who wants a place to live, this would have been solved a hundred years ago.