As an example, Habitat for Humanity wants to build 6 stories of below-market-rate condos in Redwood City, which complies with zoning codes, but are being opposed by a lawyer who lives on the block, because it's "out of character" with the other buildings in the neighborhood.
It's a byzantine approval process that includes City Councils who are 100% responsible for local housing decisions, and are elected and responsible only to current city residents. Zealous Planning Commission boards who consider things like shadows and wind when making decisions, EIR (environmental impact report) appeals, CEQA lawsuits, a constitutional housing amendment (Prop 13) that incentivizes existing homeowners to oppose new development, and more...
(It help if more areas were upzoned, but it's not the reason by any stretch)
On the other hand, you don't want to end up with this[1], so considering things like shadows and/or view obstruction has some merits.
1 - https://en.wikipedia.org/wiki/Vera_Coking#/media/File:View_o...
I haven't checked housing costs recently, but anecdotal data (from people looking for apartments, etc.) is that prices are still through the roof (so to speak). It could be prices will decline later this year or into next year, but pretty soon the supply of new housing will cease to increase so rapidly.
It remains to be seen what happens to housing costs. I could be all wrong, but I'll predict there won't be a significant decrease despite the rapid pace of development.
I had a friend a few years ago who -- unemployed -- drove across the country to Portland, took a job fixing inventory at a music store, and got a room within walking distance of the train. Try pulling that off in the Bay Area!
The solution IMO is a "trigger" law that will increase zoning heights in a suburb when the nearby suburbs pass similar laws. This prevents any one town from becoming gentrified if they raise their growth rate above the regional norm.