Most people don't care about zoning, but they really do care about zoning changes that affect them.

Being told you can't live in your house anymore because it is now zoned commercial gets people riled up, and being told the same because it's now too expensive does the same.

It's a hot-button issue but you can defuse it by putting safeguards in, though they're not always successful - see eminent domain scandals.

Actually this entire construction is wrong: currently people who improve their land get penalized by higher taxes. If there are only land value and not property taxes, we are in fact increasing your freedom to make changes to your land without penalty.

Your argument ignores that changes to neighbors' property would then cause an increased burden to a land owner. This would not seem to be an improvement for freedom