I think it's because San Francisco was "progressive" back in the 60s, which means that there's been time for the past couple generations of progressives to become entrenched NIMBY's with a vested interest in the status quo while still maintaining their "progressive" identity. It's a very weird yet oddly lovable city.
Therefore, if you're worried about protecting the security of renters (who constitute about 2/3 of residents) and especially low-income renters then unless you know any better the predominate political narrative is to oppose new development.
The narrative is deeply entrenched. For example, the narrative says (among other things) that new development destroys rent controlled properties as any building newer than 1995 is exempt from rent control under state law. But rather than lobby to change the state law to make it easier to expand rent control (i.e. some sort of quid pro quo--expand rent control at the same pace as new development so total rent-controlled units stay the same) people just reflexively oppose new development.
Because homeowners have co-opted the disparate opposition groups, the potential for compromise and novel solutions is stonewalled and never communicated through normal channels (community groups, neighborhood papers, ballot initiatives, City Hall backrooms, etc). All the voters know is to oppose development. Full stop.
Affordability and gentrification are interrelated, but so is quality. I live in a relatively low-density neighborhood of mostly 80-110 year old houses. Lately, we're getting new development in the form of cheap older rentals being torn down, and large-for-the-lots (typical lots are 1/10 acre) new houses being built. These new houses are considerably more expensive than neighborhood average. My above-average house is worth about $300, but a new house two blocks away just sold for $475k before construction was completed. So I have concerns. On the other hand, that new house replaced a run-down rental. A lot of these old houses simply have worn-out bones and will never be good again. Left on their own, they'll contribute to the neighborhood drifting "bad", with cheap rentals.
And developers? There's no money in building "affordable" housing. They'll build upscale if they can, because it's much more profitable.
What do I want? A better neighborhood, yes, but one where the people who already live here can continue to afford to live here.
The key issue here is that this cycle works best in the long term, but can cause pain in the short term. Additionally, if the new housing doesn't match the complete incoming demand for the neighborhood, the prices of the old units rise as well furthering displacement. In turn, she advocated for the removal of zoning, for as it stands right now, there is an extreme demand for housing in neighborhoods that are zoned in such a way that restricts their density from matching the demand in addition to pushing commercial activity to fixed locations where prices will always be high due to demand to living within walking distances to stores.
There's deregulation; deregulate anything that isn't explicitly for safety purposes. Schools and sewers and congestion and all of that should ideally be paid for with impact fees, not reflexively blocked out of hand. Every additional regulation is a cost on the developer passed down to the owner and the renter, and with enough small costs eventually you start pricing people out.
There's also more regulation; a lot of the current appreciation is because America places a lot of strange emphasis on housing as an investment product, rather than a commodity. It wasn't always this way; go back a few decades and property performs more like a bond than a stock, none of this speculation and house-flipping BS. And in most housing markets, housing works this way. Tax or restrict purchases of non-primary homes and such.
And then there's the final thing; lot size regulation. You can cap how big a lot can be for a project of a given size. You could build a very comfortable house at 1/10 an acre, or even 1/20th of an acre; it's hard to justify a single-family house much bigger than that. And in general, buildings should take up at most a quarter of a normal city block.
https://www.strongtowns.org/journal/2017/8/1/how-luxury-hous...
http://www.sightline.org/2018/01/18/every-month-portlands-in...
YIMBYism is more center-left than progressive.