Are you suggesting that if cities upzoned more areas for smaller dwelling like townhomes and duplexes, families would, what, refuse to live in them? Those units would just remain empty? Because if so, you're wrong.
It's not the free market, it's not individual choice blocking smaller dwellings, it's zoning regulations.
The popularity of detached housing is really only a post-WWII phenomenon created by massive infrastructure investments in highways and tax incentives for homeownership (versus renting, which is basically pouring money down a hole in the US). It's a historical anomaly, and I don't think you can disentangle the desire for detached homes from the policy structures that incentivize it.
But saying [t]he popularity of detached housing...[is] a historical anomaly is somewhat misleading.
Widespread prosperity is new by historical standards. Penicillin is new by historical standards. Air conditioning is new by historical standards. It's unclear why owning detached housing - similar to what only richer people used to own - is any different.
To talk about efficiency, it's also not very efficient to assign a whole dwelling to each family. But that's what people want to have, so you can either plan for that, where you at least have the possibility of success, or you can continue to plan cities based upon utopian fictions, and continue to get the same results.