To talk about efficiency, it's also not very efficient to assign a whole dwelling to each family. But that's what people want to have, so you can either plan for that, where you at least have the possibility of success, or you can continue to plan cities based upon utopian fictions, and continue to get the same results.
Are you suggesting that if cities upzoned more areas for smaller dwelling like townhomes and duplexes, families would, what, refuse to live in them? Those units would just remain empty? Because if so, you're wrong.
It's not the free market, it's not individual choice blocking smaller dwellings, it's zoning regulations.
The popularity of detached housing is really only a post-WWII phenomenon created by massive infrastructure investments in highways and tax incentives for homeownership (versus renting, which is basically pouring money down a hole in the US). It's a historical anomaly, and I don't think you can disentangle the desire for detached homes from the policy structures that incentivize it.
But saying [t]he popularity of detached housing...[is] a historical anomaly is somewhat misleading.
Widespread prosperity is new by historical standards. Penicillin is new by historical standards. Air conditioning is new by historical standards. It's unclear why owning detached housing - similar to what only richer people used to own - is any different.
Increasing road capacity can cut commute times, for like a week. When people know they can get somewhere more quickly and more reliably because of increase road capacity, they drive more, which in aggregate, causes the new roads to become as gridlocked as the old roads, and the old roads are still gridlocked.
Single occupancy vehicles are just not efficient for moving large numbers of people around. In order of preference you basically want people to 1. not have to commute (telecommute) 2. walk 3. ride a light vehicle (bike, electric scooter, whatever) 4. take a train or Ferry 5. take a shuttle 6. take a bus 7. carpool 8. drive 9. take a plane.
(Actually not sure where Lyft and Uber fall in that stack given there's evidence that ridesharing platforms seem to be increasing traffic in the areas they operate in.)
Of course you can mix any and all of those but your city's transportation infrastructure ought to accommodate each of them and prioritize each in kind.