The problem is that there's no effective way with (currently constituted) zoning law to 'trade' the externalities of density. You can't compensate your neighbours for shadows or traffic or other impacts in exchange for being able to build larger. So the choice faced by most residents is:

1) Acquiesce to high-density development, and receive zero compensation, or

2) Fight high-density development, and probably get zoning laws imposed at minimal personal cost.

You can see why this setup incentivizes people to push hard for "static neighbourhood" zoning law.

Being near a city core you should EXPECT high density development.

Being near a satellite nexus area of such a city, you should expect medium density development around the nexus of civic infrastructure.

Japan got how to do urban planning correct.

Yes, Japan got it much righter than most.

Or:

3) Agree to high-density rezoning, endure the construction noises and shadows and whatnot for a couple of years, then sell your house to a condo developer for a massive profit and never have to work again.

This isn't a hypothetical, I saw it happen on the street I lived on in college. The detached houses fell like dominos and were all replaced by high-rises until there were none left.

Sure, if you want to sell. But if only some of the people want to sell, then the problem of non-compensated externalities arises again. It's a classic collective action problem arising mostly from the fact that things like air rights, street space, and so on, are barely tradable.

I disagree; being in a neighborhood with high density zoning allows property value to rise precipitously.

There are plenty of exceptions, I'm only talking about the most probable outcome.

This also doesn't address industrial building or other nuisance construction.

If the HOA is a corporation with the residents as shareholders, then Walmart or whoever can go to the council and offer to buy out the area or make an exception. The local neighborhood can vote on whether to allow it to go through, or ask Walmart for money to compensate them for the disruption. There's very little friction if Walmart is willing to come to an agreement with the neighborhood. The worst case is that Walmart buys the HOA corporation, and there's always some amount of money that will buy people off.

If the local city manages it, then it's a long drawn-out process of lobbying for zoning controls that apply city-wide. And offering the citizens money to vote for it might be illegal in some jurisdictions. Even if Walmart and the local citizens are okay with it, a couple hold-outs or people from other neighborhoods can delay the entire process by complaining to the zoning board.

> the affected neighborhoods would have HOAs by contract limiting building freedom

Then I wouldn't sign those contracts or I would not move into their properties. I'd move to the outskirts of their neighborhood and build there.

Governments have a perceives authority to enforce their regulations on everyone. No one perceives Joe-blow as having the same power. It is because of this that society can keep people in check when these situations arise but no one can control the government as they are presumed to have abilities ordained allowing them to (unfairly) arbitrate these situations.