Most things are accidents of time/place/whatever. Including whatever resources those newcomers are bringing to a city. Is migration and density growth a strict good? A strict bad? Somewhere in between? Lots of ink have been written on that, I'm fairly personally neutral, but I'm very skeptical of the "this is a simple problem with a simple solution" crowd. I'd love to see a decentralization of high-end high-paying jobs. Right now if I wanted to live back near my family, I'd have to work on problems that are less interesting to me for quite a bit less money.
There's nobody exactly being unselfish in this debate... People who want more density or to save their own money argue one way. People who want to live a (different) way argue another... it's basically all wants considering we have so many other cities in the country with plenty of space.
The way I look at it is that the incumbents are there due to mostly accidents (of birth time/place) or of luck. Why should newcomers be disadvantaged due to accidents and luck?
Land is a finite resource, desirable land even more so. This is why we have things like property taxes, to reinforce the idea that property ownership is a privilege, and you have to pay for that privilege by contributing to the common good. We have zoning laws to attempt (though we often fail) to collectively make the best use of the limited land we have.
And from the other side, I just see it as just incredibly selfish to oppose reasonable housing development because people don't like change or are afraid that their home's 400% appreciation in value might drop to only 300% or whatever.