Being told you can't live in your house anymore because it is now zoned commercial gets people riled up, and being told the same because it's now too expensive does the same.
It's a hot-button issue but you can defuse it by putting safeguards in, though they're not always successful - see eminent domain scandals.
But this is more of a straw man than a real objection; as there are various and sundry ways to resolve this (for fixed income, you can have property tax deferral until sale-after-death, you can have homestead exemptions, etc etc).
I fail to see why that’s a problem. Those taxes go to fund local services (like schools, police, fire, etc.) whose costs likely scale when the local cost of living and land values scale. Buying land doesn’t mean you should get to lock in the price you pay for these services for perpetuity.
Having un-utilized land in your family generation after generation codes as noble, but economically this is much worse than owning an AirBnb or a vacation home or being a Chinese investor who collects vacant luxury condos, or any of the other boogeymen of housing discourse. These people hold one unit off the market, and in the meantime they maintain it or finance its construction. The landowner builds and holds wealth for nothing but parasitism.