Maybe you should look at your house as a place to live rather than an investment vehicle.
Or maybe you should just accept that investments come with risks.
The exact same thing that makes me think I can't just walk into your home and take what I want, because I want it and you're not using it. The rules set forth by the society we live in, as decided by the people living in that society. Sure, not all the rules are great; many of them are awful. But the solution to that is to fix the rules, not to say there shouldn't be _any_.
No, it doesn't work that way. My house is not your house. My land is not your land.
If you're going to restrict my use of my property to increase your property value, you should pay me for that privilege.
The rules in many areas used to forbid selling houses to black people.
Those rules were changed. By force.