Most of the places where houses are already so cheap that the price of actually building a house is a blocker for affordable housing typically have a large glut of unwanted property from population drained to other areas.

If I had to guess it's not so much the actual cost of building the house as it is the profit margin of those who are building homes. Why build a $200,000 "starter home" when you can build a $478,000 home with gigantic margin on "luxury" trim like cheap granite countertops and "hardwood" floors for roughly the same cost? And then once you get the ugly design down you can just copy and paste it with few adjustments and save even more money.

Unfortunately we have bad incentives aligned here versus stated goals.

Manhattan and San Francisco are like Japan and Argentina in economics. They're just not the same market as most places. In Manhattan in particular the value really is the land.

Cities still hold a large portion of the population of the US and a lot of jobs can't just be done remotely even if employers actually adopted remote work which many really really don't want to. Housing isn't just a problem in Manhattan and SF it's hitting cities and suburbs everywhere.

Go tour some unincorporated land in the Southwest and you'll see how cheaply homes can be built. The problem is that once you're spread out enough to be able to do that, the resources required for daily living have gone up significantly because they're no longer amortized over population density and time. And that same gradient applies to suburbs/ruralish areas with building codes.

Building a bunch of housing in the middle of nowhere for unhoused people isn't going to work unless you also provide them heating, transportation, etc. The least expensive way of doing that is to make it dense like a city. At which point you're just building a planned city, which is fine, but not particularly in the purview of the US.

In my modestly sized city nowhere near California or New York the problem has nothing to do with building costs and predates the rising costs of building materials. It has everything to do with poor city planning and a lack of building permits for middle income housing. Plenty of over priced McMansions and plenty of half occupied apartments. Not enough normal houses. So the market became irrational. People buying houses site unseen even before COVID. And property values going way up.

Another contributor to the problem was outside speculators buying residential properties. Specifically Berkshire Hathaway and Zillow come to mind. But I'm sure others were doing it too. It should be illegal.