Because land has fixed supply, as you've said, the tax is perfectly efficient: it causes zero deadweight loss. But LVT actually has positive externalities as well- namely, it makes it nearly impossible to speculate, which with land consists of buying land and holding it off the market unused or underused in order to sell it later. Speculation is very inefficient and also quite common, so removing it actually makes the economy better than it was before an LVT.

As for the tax being too high, if people started to vacate land that would mean the tax rate is over 100% of the land value (if it's at 100% people won't vacate, they won't be able to extract rents from it but it won't be an economic loss). Governments have every incentive not to tax land over 100% value because they actually lose money from doing so, so that's a pretty good security that it won't happen.

I think I need to think about this some more. It seems like the goal of government is to figure out the tax rate that counters the land value perfectly without taxing any more. The fact that it is unimproved land value means we aren't stuck in an efficiency trap since a diverse enough set of uses can improve the land sufficiently to generate a reasonable economic profit in non-rent form. Countering speculation is definitely a strong positive effect.

I worry that the overtaxation causing vacation mechanism isn't very efficient because if the tax takes a small part of the land value improvement it seems that the government gets additional unearned revenue while only land with little to no improvement would be vacated. I think people only vacate if the taxes cut into their additional revenues too steeply. I'm not sure exactly where the too steep point is, but I think governments would have an incentive to overestimate the value of land.

That is not the sole purpose for the public collection of land rent. Land rent is based on demand for land. Land rent is what you would pay for a particular plot vs. some other plot that is available.

The question is to whom should this payment go, and "fee simple" is an entirely inappropriate mechanism for payment.

Everyone has an equal right to the use of land, limited only by the equal rights of others to use land. What the state does not need for the provision of public goods & services should still be collected and returned to the people on a per capita basis, as a simple matter of social & economic justice.

Overtaxing is definitively a problem but I personally am not a fan of extremes. Even just 50% of the ideal tax rate would be sufficient to significantly improve an economy while not ruining anything by being overzealous. There will have to be an escape hatch so that people can defer taxes for 5 years.