The roommates part seems essential. There are a number of reasons but if I had to guess:
- With roommates you can often take advantage of someone who has been living there since before some recent price spikes (especially in certain neighborhoods). If you find a place by yourself, the landlord is free to charge whatever they want unless you can find a rent controlled apartment, but those have multi-year waiting lists.
- Vacant units are often renovated so the rent can be raised.
- Economies of scale. If you live by yourself chances are you want a place with your own kitchen, bathroom, etc. These are shared spaces in a roommate situation.
- There seems to be more competition for single-residency housing. I have no data to back this up, but just from talking to people it seems that there are more people who hate living with roommates than who hate living alone.
I am not very familiar with San Francisco, so I left myself some wiggle room ("(at least in most of those cities)") but having a roommate is not a requirement for anybody making a tech salary and wanting to save money in those other cities.
Because they value different things than you do; whether that's the food, the theaters, the museums, the climate, or the social culture, or something else. Different ones of those for different people.
> It's one of the most depressing place I've been to, it's like those dystopian movies...
Other than in the generic ways that most urban environments are "like those dystopian movies" to someone whose lived most of their life in a suburban environment, I don't really see it.