Artificially suppressing property taxes to protect entrenched owners is a lot like taxing the internet to support newspapers and record stores.
Currently, why do people buy real estate as an investment? It's not because they believe the actual building materials will become more valuable, but rather that the land will. We see far more of "land banking" than speculation in commodities, insofar as commodities tend towards marginal cost.
Anyway, if a land value tax is in place, the land value increase won't be captured as profit, and thus all locations will only be used by people who mean to use them.
There is no reason to believe that middle class residents will be "chased out." However, we would see market incentives that would lead to gradual densification in residential areas that are encountering rising demand.
Then you won't have people hold on to inventory just for the taxes, and eventually things will even out as people sell each year.