Regulation is supposed to prevent or limit those externalizing costs. However it's nearly impossible to put value on something like that.
Could you elaborate? I mean, the common law is littered with examples (stare decisis) throughout the industrial revolution which establishes strict liability for, say, someone else's right to enjoy their own land or property. In the UK, this extends to a tenants right to peacefully enjoy his home, even when he doesn't own it.
I think municipalities that can support high density development are in short supply. High rises are just concrete and rebar. But if there's no transportation, not enough energy, security, etc - then no development will take place. We've left those roles to local government, and I think we should challenge that decision.
The supply of land rebar and concrete isn't that scarce (thanks capitalism). What's scarce is the organization and infrastructure that can sustain a town - and the limited land around it. If there's something that needs more capitalism it's city planning.
It's also very easy to find negative effects of displacement. The one that springs to mind easiest is that old people who get displaced die many years earlier. These negative externalities fall on people who simply don't have the money to pay enough to cover these downsides.
For a totally trivial example with real dollars: moving costs are non-zero, so when people move against their will we trivially destroy value there.
I disagree with the sibling comment which claims that the existing community/neighborhood have done is what drives gentrification - a LOT of gentrification is driven by a balancing act between price, public transportation and safety. So, I would argue that for the gentrifiers, one neighborhood is often as good as any other, but for the people who have lived there a long time, that specific neighborhood is what is important. So in this sense, it is value destroying to push these people out on a 1:1 basis.
Often gentrifiers aren't even moving to their first choice neighborhood and are merely priced out of the places they actually want to live.
For instance, people want to live in city x because this city have nice features (good public transport, good public spaces, buildings size that leave to see the sky..). Those features are in some cases due to location but, in other cases, are due to what the people that have been living there for generations have done with the city.
Now outsiders come and out-price the people that "created" the city and, in the process, destroy the value of the place. So, people start to move to another nice city. Rinse and repeat.
This, it seems to me, is not a "fault of the market", but the effect of an optimization process run amok. And I have not even mentioned speculation.
But I think housing is definitely a thing capitalism struggles with because it is a good everyone needs and there is a finite amount of it, so much of it is zero-sum allocation.
Obviously I think we need to invest more in the non-zero-sum parts of it, but it's also a difficult political problem with lots of common goods and externalities.