Well this is no different than hunter gatherer society where land was 'owned' by those willing able to defend it. If your tribe couldn't defend your land from invading tribes, the other tribe would take ownership of it.

What you're ultimately complaining about is that other people exist, which is often noteworthy of complaint, but you'll be hardpressed to find a solution.