For one thing, there was (and is, in, say, China now (can I get some more commas in here?)) massive inequality between the wealthy, electrified urban areas and poor, low education, rural areas. This is a problem for a democracy like the US, where an urban majority (which only happened in 1910-20) cannot run rough-shod over a rural minority. That political problem is enhanced by a media who is certain to show the famine out in the sticks, in comparison to the comfort of your favorite city.
Further, all of the resources are in rural areas. You'd have a hard time digging coal out of Appalachia if every moves to Jersey City. And you still have to deal with agriculture: your bread is coming from the Dakotas and your corn flakes from Kansas. Unless, of course, there's no infrastructure to transport it from there to here. And this was before the "green revolution" made food really, really cheap.
There are also other factors. The US was largely founded on the idea of a homeowner, living on a homestead that he owns, with property rights and all that. And then there's the belief (which I kinda agree with) that people who own property are more involved in their communities than people who don't, and thus that home ownership is a good thing in itself. ("The city council is doing something stupid! Do we go down and yell at people, or do we pack up the apartment and move?")
But consider as just one example how much the population working in agriculture has decreased over time - much of that was going to happen whether roads and bridges and power lines were subsidized to go out there or not. When this is private investment, it's fine. If it fails, some company goes bankrupt, life goes on. Unwinding money-losing public infrastructure is much, much harder.
The last paragraph, that renters aren't politically invested, I think depends a LOT on the ability of people to become owners in the first place. Ownership is more the effect of someone who has a certain amount of wealth and stability in their life (thus the resources and desire to own), rather than something that CAUSES wealth and stability in your life.
So as for the idea that "the country was founded on the idea of the homeowner," I understand, but I think we have to be careful not to think that it somehow makes an owner morally superior to a renter. That just leads to discrimination against people who probably would love to own but don't have the resources, and thus don't need the deck further stacked against them.
I'll try to explain, giving a US-centric narrative because that's the one I know best, but similar parallel things have happened around the world.
- - - - -
Mass urbanization started in the 1800s and accelerated in the 20th century as the world transitioned from a rural/agricultural economy to an urban/industrial economy.
In the US you can see how this worked as placed like New York boomed, but also smaller cities like Buffalo, Cleveland, and many others.
This alarmed a lot of people, notably FDR. They didn't like that people were leaving smaller towns and rural areas and flocking to cities, and he wanted to try and keep the population more spread out. But they were just doing what was economically rational, getting better jobs in the city. And this is still going today -- see China, or the article on HN the other day about entire abandoned Spanish hamlets for sale).
A lot of the push back against urbanization happened via regulatory policy and infrastructure investment. A great case of this is the Rural Electrification Act in the US.
It's cost-effective to serve electricity to a city cheaply from a central plant, thus cities over a certain size (critical mass) started getting power plants and cheap electricity, whereas places below that threshold did not. There was actually a lot of innovation going on in this area, with "off grid" generation and low-power appliances etc. to serve the large, distributed rural population, but it was lagging behind the "convenience" of city power. So the US embarked on a government led program to mandate electrical service be provided to rural areas. The history of all that is complicated and I won't try to narrate it all here, but you ended up with a ton of messy side effects -- see all the land served by PG&E and the California wildfires today.
The same thing happened with roads and bridges and basically every other kind of infrastructure. Without these programs people were trending to concentrating in the larger towns around the countryside (above critical mass) and depopulating rural areas, and the programs slowed that down a lot by providing heavily subsidized infrastructure to make their life more comfortable.
At the same time a different movement was going on in cities, motivated for many reasons, to start limiting development. First it was to separate out polluting industrial factories from residential development -- arguably a very good call -- and then it spread to limit development generally and prescribe lower and lower limits to density. This was greatly aided by that same push to bring city utilities to rural areas -- the interstate highway act, for example, meant that suddenly there were vast areas a bit outside the cities that had great road access to the city center and high voltage lines etc. already going out to the farmers. Suburbanization here we come.
Well, fast forward a few generations and the economic reality of much of that infrastructure is hitting home. Why don't we build big infrastructure projects like we used to? In part it's because a huge amount of maintenance money goes to keep the rural roads and power lines operating at huge losses, to places that would probably either have been abandoned or have adapted to an alternative "off-grid" set of technologies if not for the massive infrastructure subsidy they receive.
But it's more pernicious than that, because the "spread everyone out" policies resulted in a pattern of development where most of the _urban_ areas are also below the critical population density where infrastructure has a positive return on investment. Ie. most suburban development is also an infrastructure money-loser.
These policies are becoming increasingly difficult to sustain, resulting in deferred maintenance and generally bad infrastructure across most of the land area, and also a general lack of funding to keep up with the population growth in the major cities.
And then at the same time, restrictive development policy - which is mostly concentrated on housing, not commercial development - means that housing in the cities that have the best economic opportunity for people is increasingly scarce and thus increasingly expensive.
A final tie-in to your point about financialization and financial engineering -- the increasingly Byzantine development regulations mean that it takes higher and higher skill and more and more patient capital to negotiate the development process. Whereas it used to be quite common for a family to do something like split their lot, sell it, and an individual to buy that lot and build a home on it -- now development activity is increasingly limited to high-capital sophisticated firms that can deal with the regulation and holding cost.
You also have the government funding this system via Fannie and Freddie etc., leading to the securitization of mortgages and a shift away from local savings and loan type funding. That, plus a whole generation of individuals and institutional investors believing that "your home is your best investment," so you have a giant coalition with a vested interest in housing prices steadily climbing. So it's not too surprising that you'd get a lot of financial engineering effort to try and keep the system steadily growing and that this could lead to bubbles every so often.
- - - - -
Whew. That's an attempt to summarize over a decade of reading and observation on my part, so I apologize it's rather long and rambly. But I hope that gives you some better illustrations for how this whole thing seems interconnected to me.