It only seems that way sometimes in SF because there is literal decades of pent-up demand that needs to be satisfied, resulting in people willing to pay quite high prices.
Neither is the supply of land.
The question is, can San Francisco build enough to satisfy this demand, without replacing so much of the existing housing stock that San Francisco doesn't look like San Francisco any more? I don't know. And maybe it's not important. Cities do evolve, after all.
But I do think the question needs to be asked, and debated.
The answer is: yes. The heart of San Francisco lives not in a brick or a stone or a board, but in the hearts of San Franciscans. Gold in peace, iron in war.
Even if some new luxury apartments are pied-à-terres which would not be in SF in the absence of new construction, if some are actually occupied then rents for the city in general will also benefit. (New downtown high-rises with super-fancy penthouses can easily have many real residents on the lower floors.)
The only guarantee one has is that developers will build what generates the most profit for them. That's basic economics. If the most profit is found in housing for locals, then locals get housing. If it's in pied-à-terres, then they don't.