Fuck those people - they don't deserve community nor the benefits of it they worked for, because they can't afford the prices you are willing to pay to kick them out (despite most of them being decent citizens with jobs and everything).
It's kinda messed up. Not saying I have a solution, just pointing out a real problem.
Honestly, you are saying that the only consideration people should have for one another is net worth and that interpersonal relationships are of no value, yet you are angry that your lack of those same things could be used to keep you from spending your money (suggesting value). This is a pretty strange.
I do think that perhaps some consideration besides "number of dollars" should be taken in situations like this, because I don't really think a group of people should be able to so fundamentally fuck over another group because they can afford to. But those are my ethics - I value respecting people on more than money.
I actually think the car analogy is useful. Not subsidizing housing in San Francisco does not deny people the ability to start families. They can still start them in many other fine places that they can afford. Not driving a BMW or Tesla doesn't deny people the ability to get to work. A Honda or Toyota does that just as well.
NOTE: "Affordable housing" does not apply exclusively to people who have lived in the area for N years previously as far as I know, and even if it did I don't see why that would be fair. If you rent, you rent. When your lease is up, if the price goes up, that's how rent works. Also, there are already rent control policies that would serve this purpose.