Bay area's housing shortage is entirely self inflicted by its residents who refuse to allow more housing to be built. All this to maintain some arbitrary idea of character. A character that is frozen at some random point in time.

There is no good reason for most of the bay area to be strip malls apartments and 2 story apartment complexes. There are more than enough examples of cities(Barcelona, Paris) with character, history and more vertical housing.