It's all relative. If everyone else moving in can spend a mill plus, you're priced out. If you and your neighbors happen to be the only ones improving things, and the mil plus group comes in and stagnates/displaces you through gentrificative forces, requiring you to to start from scratch elsewhere.

Lets say you live long enough for the cycle to repeat multiple times. I assume proponents would say, "well, hooray, everyone but you is way better off, I guess it sucks that you weren't better at managing money", when all anyone ever wanted to do was live together and be left alone.

> all anyone ever wanted to do was live together and be left alone

It's a great ideal, but the problem is that land is a finite resource, so eventually you'll run out of space that is necessary for the next group of people who want to live together and be left alone to actually do so. At which point, you either have a stark divide between those who got in their claim and those who don't, or you come up with some redistributive system that gives newcomers a chance.