Well, but in the absence of an overriding rulership, i.e. anarchy, you actually can do what you want even if you're restricting the individual freedom of others and denying them access to the same means and rights as you do. There's clear motivation for some people to do that, and what's going to prevent them from doing that to weaker or less resourceful people? And a system capable and intent on effectively doing that essentially becomes a de facto system of governance, and the situation ceases to be an anarchy.