It is all about perspective. Compared to London, Paris, Tokyo, New York, most of the bay area is incredibly low density. Huge chunks of SF, Oakland, Berkeley, Palo Alto, Redwood City are 1 or 2 story single-family homes with huge lots. Regulations prevent even a modest 10 story condo.

You don't have to make it into a false dichotomy of LA-like insane urban sprawl vs. the current situation.

The Bay Area's density is a fraction of of New York City's density. Yet rents in the densest parts of NYC are even higher than those in the Bay Area, which suggests that many people want to live there.

You're sort of both right. One reason I don't want to relocate to the Bay (despite several offers) is when I do the math, my quality of life is substantially higher here in the middle of nowhere New Mexico on astonishingly less money. If you're not after the glory, it doesn't necessarily make economic sense. Doubly so if you don't like the kind of work that exists in the bay. If the housing and commute time issues weren't so dramatic, I would think about it, but still probably would stay out here.

If you are single or DINK (double income no kids) - moving to Bay Area (or NYC for that matter) can have economic sense.