Under the assumption that everything you have and get is just and fair I agree. It is a strong assumption though, and It deserves to be examined.

Externalities: Do you contribute to pollution? If so should you not then pay a fair price in compensation to us all for that privilege?

Rule of law: Do you derive rent, or benefits, from property? Are you not dependent on society to define those property rights, and enforce them? Should you not pay a fair price in compensation for that privilege?