[1] - https://en.wikipedia.org/wiki/Social_contract
To refuse the social contract is to accept natural law (law without an organized government with a monopoly on violence). If too many people decide the social contract is not in their favor, they will do this by noncooperation or overthrowing the government. Under natural law, people are not prevented from building shelter, freely gathering food, etc. This is why people believe it is unjust and illegitimate to hold property rights above basic needs. Property (or at least more property than you can personally protect) is essentially a construction of the government, and freedom to gather and grow food is included in natural law. People who are deprived of important natural freedoms are likely to refuse the contract.