If a tax on holding onto unimproved land is nearly zero, we're very likely to implement zoning laws banning development, because citizens have little reason to change things. But if the holding cost increases, (say with a significant Land Value Tax) folks now have a reason to legalize development.
...Californians should consider the role of Prop 13 in its current housing crisis.
Artificially suppressing property taxes to protect entrenched owners is a lot like taxing the internet to support newspapers and record stores.
Currently, why do people buy real estate as an investment? It's not because they believe the actual building materials will become more valuable, but rather that the land will. We see far more of "land banking" than speculation in commodities, insofar as commodities tend towards marginal cost.
Anyway, if a land value tax is in place, the land value increase won't be captured as profit, and thus all locations will only be used by people who mean to use them.
There is no reason to believe that middle class residents will be "chased out." However, we would see market incentives that would lead to gradual densification in residential areas that are encountering rising demand.
Then you won't have people hold on to inventory just for the taxes, and eventually things will even out as people sell each year.
Building costs are not a complete explanation for the current astronomical prices. Political choices to restrict construction play a huge role.