Being told you can't live in your house anymore because it is now zoned commercial gets people riled up, and being told the same because it's now too expensive does the same.
It's a hot-button issue but you can defuse it by putting safeguards in, though they're not always successful - see eminent domain scandals.
But this is more of a straw man than a real objection; as there are various and sundry ways to resolve this (for fixed income, you can have property tax deferral until sale-after-death, you can have homestead exemptions, etc etc).
I fail to see why that’s a problem. Those taxes go to fund local services (like schools, police, fire, etc.) whose costs likely scale when the local cost of living and land values scale. Buying land doesn’t mean you should get to lock in the price you pay for these services for perpetuity.