The same place we do, natural law, note that under natural law we can do whatever we please. The underpinnings of non-fiat social contract is the idea that we give up certain rights to the state, namely the right to commit violence against others, in exchange for protection of the remaining rights.

In all honesty though we get our rights from wherever we can convince others that we have those rights, for example in the UK if you're born to the right parents you have the right to live in buckingham palace. They claim those people get those rights from God.