The short version is this: Property owners now have shocking amounts of control over what their neighbors do with property. Owners, especially of single-family houses, elect officials who restrict development through zoning and similar means. Height limits and parking requirements effectively mandate single, detached housing in most of the U.S.

This only really got started in the 70s (see http://jakeseliger.com/2013/07/03/jane-jacobs-is-everywhere-...), and it didn't get really bad until the 2000s, when the shift back to cities ran into insane urban zoning rules to produce huge affordability crises. By now, most developers who dare to build condos or apartments have to build luxury apartments: http://www.wsj.com/articles/rents-rise-faster-for-midtier-ap... because that's the only way to make the economics work.

In the meantime, much of the population growth has shifted to Sun Belt cities in Texas, Arizona, Georgia, and Florida where development is easier and/or simply sprawls more.

If you want more economical, why not just have everyone live in barracks?