Down that train of thought is "taxation is theft," and it's not a good place to be.
(Cards on the table, I've come from the other side of the story: absentee landlord with undeveloped property sitting next to a relative's home, and the neverending fight to keep the dead trees on that property from crushing said home in a storm. The process to get a property that is owned by someone five states over who is holding it to be sold for 2x the market value and hasn't seen the property in fifteen years declared a nuisance property is atrociously long.
Owning property implies an obligation to the place where it is owned. The law should have more teeth to enforce those obligations.)
But I see that as no excuse to take my property away if I'm doing everything right. And the squatters can't tell who is absent and who is just on vacation. We have to deal with this problem seriously.