> I understand this is not for everyone and I understand the negative perception of 'outsiders' pushing the price up for locals, but at the end of the day I'm optimizing for my own quality of life.

It's unfortunate that you phrase that as egoistic because I think that apart from real estate price (which is highly location dependent because it depends on zoning laws) locals are better with the influx of money. In my experience in France, some rural places with low real estate tension are very happy to see urban people coming, but in other (like small coastal cities) there is a huge resentment because real estate skyrocket in a relatively short time.