The new neighbors could come together and buy out the racetrack. Take out a mortgage to turn it into an apartment building or something, then sell the building to pay off the mortgage.
If the value to the neighbors of not having a racetrack there is at least as much as the value to the rest of the market of having a racetrack instead of an apartment building, this should be economically viable. If it isn't, isn't that a solid case for leaving the racetrack there?
But then someone else could do it. Rich investor goes around to everyone in the neighborhood and offers to buy their house for $25,000 over market, contingent on enough people (including the racetrack) agreeing to sell to make it worth their while. Then if that many people sign up, they resell all the properties for a profit now that the racetrack is gone and all the property values have gone up by $50,000 each.
The result is that the people who live there now might not be able to live there after, but that's only because the racetrack is the reason they can afford to live there now. Otherwise the property would have been more than they can afford from the start.
If, on the other hand, I've built my house next to his plant and now started to complain about the proximity of a potentially dangerous thing next to me - well, in this case Bob was there first.
And then there are all these questions for extra credit like what's gonna happen if Bob wants to put a second plant right next to his current one?
A real-life version is, Bob's reactor was scheduled to be shut down in a few years, but now he's applied to extend its lifespan by another 20 years. Should that be allowed? If so, are there any conditions under which he should NOT be allowed further extensions?
1. How many people depend on Bob's reactor for power?
2. If Bob shuts down, is the replacement something like solar or a coal plant?
3. Design/safety/longevity.
If it's the same entity that allowed those preschools to be built next to a nuclear reactor, then absolutely not.
But let's change it. Let's say Bob is running a rather large nuclear reactor. Initially this is fine - nobody is around him and nuclear power is very environmentally friendly. However, time goes on, and after a few years Bob and his large homemade nuclear reactor are surrounded by hundreds of brand new but very full preschools all bordering his property.
Do we have a right to ask Bob to stop running his nuclear reactor? Situations change, risk models change. What was originally fine is now a hazard to other people.