Based on the relatively small number of billionaires/rich people in the world, that's a compromise I'm willing to make. If the rich want to house the poor they can do so through a charitable foundation.

Furthermore, as has been pointed out already, we could make it so that only properties that sit within a predetermined land mass size would be exempt from tax.

Why go by size and not by value? Someone having a huge estate in the middle of nowhere is not a problem, vs occupying the same amount of space (or even a much smaller, but still large one) in the city centre is.

Value is open to fluctuations, size is not.

A large estate in the middle of nowhere is still potentially wasteful. For example, could that land not be used for farming or wildlife? How many bathrooms does one person need?