The problem of "free to leave" is that that only works with unlimited resources. Why should I allow a random person into my house to eat my food and promise to assist, only to be free to leave at any time. And they have to go to another house, why would someone there do the same.

Being altruistic is great, but at some point you run out of necessary resources. And altruism dies at scale when resources dry up.