> I move to a cheap, small town. I embed myself in the community and help build it up over decades. And then, at the peak popularity, in part due to my presence, I am forced out, because someone who has never been here says that they would like to live here.

Why would you be forced out? Yes, taxes on the underlying land would increase because it is more valuable, but you would be able to move into a new property in the same community, as housing would increase in density. Now if you don't want to live in a denser community, that's fine, but you don't really have a right to prevent the densification.