It could instead leave you on the hook for increases in the desirability of the neighborhood which you have no say in. So it would mean that homeowners get screwed even harder if their neighborhood gets gentrified.
The steelman for it is that society should incentivize denser development of more desirable land, by disincentivizing maintaining sparse development of it.
Meaning, their land value and therefore wealth has gone up. Many people would love to be "screwed" like this, and in fact very hard at getting into the position amenable to "screwing."
As a class, real property owners are wealthier and have means relative to people who don't own real property. The idea that they're some oppressed underclass is ridiculous.
That is exactly what a land value tax does, in a negative way. It disincentivizes buying land in a desirable neighborhood and then doing nothing with it, waiting for prices to rise.
With capital value taxation (the other alternative), a person doing that pays less tax than someone providing buildings for people to live in or businesses to operate in. So they can do it for longer.
Because any buildings on the land are not taxed, construction is not disincentivized. Well, it wouldn't be, if permitting were not insane...
A tax purely on the land has the same justification as a windfall tax. The value goes up or down independently of any action of the owner. i.e. They didn't do anything to deserve the increase in their wealth. The value comes from what the land is surrounded by.