(If you have evidence of urban areas that have allowed lots of construction during periods of high population and job growth - cities like Dallas, Houston, or Atlanta - and have also seen high housing prices, please share it; I'm genuinely curious).
Part of the problem in the bay area is that there is minimal cooperation between the dozens of municipal governments to allow more housing. None of them want to be the first or only one to do it out of fear that they'll just end up with more density and high prices. It's a bit of a prisoner's dillema.
Orange groves and strawberry fields and outdoor movie theaters and old military bases were all leveled and turned into housing tracts. But housing didn't get any cheaper - the state added 20 million people from 1970 to 2010, because there were jobs and it had the reputation of being a nice place to live.
That doesn't actually work in the real world. If you live in a desirable area, and you increase the housing supply in such a way that the area is still desirable, as soon as the price drops a tiny bit people move in from the outside and drive it back up again.