There are lots of levers that determine if a person can afford a home. income, interest rates, % income that goes towards housing, etc. But its obvious cost of building the home plays a factor.
Go compare the costs of identical pieces of plywood from the Home Depot on West 23rd Street in Manhattan and the Home Depot in Jersey City, directly across the river.
Manhattan: $76; Jersey City: $37
Is that the cost of shipping a piece of plywood an additional 3-10 miles? Is it taxes?
It's the rent.
Even if you build a "cheap" single family house that's still going to be more expensive than some equally cheaply built townhomes, or apartments because of stuff like parking requirements and minimum setbacks.
Unfortunately we have bad incentives aligned here versus stated goals.
Manhattan and San Francisco are like Japan and Argentina in economics. They're just not the same market as most places. In Manhattan in particular the value really is the land.
Building a bunch of housing in the middle of nowhere for unhoused people isn't going to work unless you also provide them heating, transportation, etc. The least expensive way of doing that is to make it dense like a city. At which point you're just building a planned city, which is fine, but not particularly in the purview of the US.
Another contributor to the problem was outside speculators buying residential properties. Specifically Berkshire Hathaway and Zillow come to mind. But I'm sure others were doing it too. It should be illegal.