Is it really a free market in places like the Bay Area where zoning is so restrictive and NIMBYs can show up to planning meetings to basically block any new construction?
It's been pointed out elsewhere in this thread that Japan also had a free market but less restrictive zoning and housing is much more affordable there.
The main argument for government efficiency would be if the government could override the NIMBYs more easily than private developers can.
Are you sure zoning is the only thing that sets apart the Bay Area from Japan?
If public housing was done on a mass scales (like it has been in the past/other countries) you wouldn’t need a lottery process, just a wait period. A lot of people are renting and can afford to wait a year or two. In the status quo we simply can’t buy at all, we forsee renting for the rest of our lives, and are at risk of not even affording that as rent goes up exponentially.
The way I see it (and the way you put it) the free market has failed to provide housing to people, to an extend that there is no free market solution going forward. If the government (federal/state/county/municipal/etc.) provides and alternative to free market housing, they will directly be competing with private housing speculators. Prices on the free market will likely go down as a result (why pay $1,000,000 for a house from a speculator when you can buy a better house from the county for $400,000). These speculators will get angry, (and some bootlickers/sympathetic middle class) but they will not have the masses behind them to generate blood on the street.
This is not an alien solution, it is battle tested as it has been done multiple times in most countries several times in history, as recently as the 1960s. The only reason not to go for it (given your two other scenarios) would be that the government is so sympathetic to the few rich that it is willing to let everything else crash and burn in just to cater to the interests of the rich.