> It won't move, but the utilization will change to be minimize the cost.

Yes people will build more medium density housing instead of having zucchini gardens in one of the most expensive regions of the world. Higher density would be good for renters in the local area.

This, taken to it's "extreme": https://en.wikipedia.org/wiki/Land_value_tax

Pretty well studied, and has a lot of support across ideological viewpoints.