This is a good point but it's not a problem with georgism per se so much as the transition to get there. There are a multitude of policies that can be used to ameliorate the change from a system based on taxation of labor and capital to that of land; a common one is giving tax credits equivalent to mortgages so that lendees are not underwater. But any shift to land value taxation will be gradual anyways, so it would be less dramatic than people imagine.