"I don't think it would be the best of all possible worlds to make the bay area look like Manhattan or Tokyo."
These two sentiments are in direct conflict though. Your options are to either build up or to build out. Building up will preserve the nearby wilderness; building out will see it turned into low density housing, roads, and terrible traffic congestion.
I know. Quoting Whitman, "I am large. I contain multitudes."
> Your options are to either build up or to build out.
Or, in a word, arcologies.
> Arcology, a portmanteau of "architecture" and "ecology", is a field of creating architectural design principles for very densely populated, ecologically low-impact human habitats.
> The term was coined in 1969 by architect Paolo Soleri, who believed that a completed arcology would provide space for a variety of residential, commercial, and agricultural facilities while minimizing individual human environmental impact. These structures have been largely hypothetical, as no arcology, even one envisioned by Soleri himself, has yet been built.
https://en.wikipedia.org/wiki/Arcology
If I had my druthers, I would roll up the low density stuff into arcologies and rehabilitate the old creeks that ran and still run to the bay. Use Permaculture (applied ecology) and passive design. IIRC the Australians were going to build a huge multi-acre combined solar chimney and greenhouse to generate electricity and grow food. &c.
https://en.wikipedia.org/wiki/Solar_chimney
We have all the solutions we need (often they are decades old) we just have to clue up and get crackin'.
And also the trend of people wanting to live in SF is a pretty recent phenomenon. The population of SF in 2000 was the same as it was in the 1950's. People were leaving SF in droves 40 years ago.
Cities can't just turn on a dime.
Other urban areas undergoing growth seem to have done much better by allowing more density. Unfortunately the California tax system incentivizes municipalities to build office space over housing. With the Bay Area's patchwork local organization and no metropolitan level layer of governance local municipalities instead play beggar-thy-neighbour as they promote only development that expands their tax base while the burdens of providing schools etc. falls to the municipality next door.
Do an image search for old timey SF and you'll see: the Sunset was sand dunes until about last Tuesday, etc. Dudes used to pan for gold out by Lake Merced.
Also, this was the weirdo town. Bohemia on the Bay. Serious folk didn't really come here to be serious and sane. The old line was, "All the crazy people move to California, and all the crazy people in California move to San Francisco." To which residents add, "And all the people too crazy for San Francisco move to Berkeley."
So yeah, you've got a young town full of nuts and flakes that no one really paid much attention to until the last ~50 years or so, whadda ya expect, eh?
Things are changing though. I was over in the Inner Sunset the other day and the beautiful people were there casting me in the shade because I look too close to homeless on the hipster/homeless spectrum.
My concern is the weakening of the commitment to keeping the wild areas of the Bay Area wild. SF is the largest city with the most wilderness around it in the world. Marin Headlands, Wildcat canyon et. al., nearly half of the peninsula is forest. I saw a coyote in someone's front yard on Junipero Serra the other day not three blocks from Stonestown Mall! In a neighborhood of single family homes, FWIW.
I don't think it would be the best of all possible worlds to make the bay area look like Manhattan or Tokyo. On the other hand, it sucks that I can't afford an apartment in my home town, let alone ever own a home, despite making programmer money. I can't imagine what it would be like if I didn't have tech skills. I'd probably be living in a tent under the freeway.