The more interesting question is, why are the 50 miles between SF and SJ reserved for sparsely packed single family homes and one story strip malls?
Redwood City is also nicely positioned as a mid-point between SF and South Bay.
Beyond that, it seems the entire El Camino corridor is being heavily developed with retail and high-density condos and apartments.
Some of these areas (Redwood City in this case) are next to major Caltrain stations with Baby Bullet stops. But they're also convenient (technically speaking) for going on 101S/N to work, etc.
That's part of why I'd prefer living on the peninsula than in downtown SF. Weather is consistently better in the summer months. More space/less people (except for rush hour traffic). More parks for recreation. Pools aren't necessarily a deal breaker for me, but it's a nice extra.
A shiny high-rise is nice but unless you can afford a penthouse or similar you're probably looking at a small unit. Less space, fewer parks, more people everywhere (tourists, homeless, etc.)
The downtown has a solid layout though and lots of room for growth. The RWC downtown has the potential for offering a lot more than some neighboring downtowns that are less dense and could be a nice hybrid between SF and more "suburban" Peninsula cities.
In case you want me to answer, there are myriad advantages: the chance to work with your most preferred tools, on a product that's more interesting you, with maximally compatible coworkers, etc. All the normal reasons people go somewhere besides where they were born for a job.
That's the only way you're going to come anywhere near your current rent. As a bonus you'll get more space generally and some community.
My unit is rent-controlled. However, the owner of the building has only increased rent twice, both being the max they could under the rent control limit. They just haven't continually increased it on an annual basis because apparently they haven't needed to.
I live in San Jose, I commute to SF 3 days a week (super thankful for the chance to work remote 2x a week). My current apartment (a 1BR I live in by myself) costs a bit less than 1k a month and is rent-controlled. The equivalent in SJ is probably worth 2-2.5x as much per month, and in SF (or somewhere in-between) it can be worth anywhere from 2.5-4x as much.
I've been hunting for an apartment more on the peninsula since I don't necessarily need or want to live right in the city, just closer (especially in case I switch jobs eventually), but I definitely want to halve the commute time at least (Caltrain plus door to door is nearly 1.75hrs each way).
But why would I want to leave my rent-controlled steal and be spending over half my monthly income for the equivalent? It's ridiculous.
So now it's a matter of figuring out the cost analysis of moving into a better area that's closer to work, and therefore an increased quality of life, or stay where I'm at where I can save much more money and have a rather crappy quality of life (I don't care for where I live now strictly other than the cheap price of the rent).