The crucial underpinning of this ideology is that We The People have given explicit permission to a specific group of individuals to have this power in exchange for freedoms we could not otherwise attain; rarely can something in life be so pure, a balance usually must be struck when considering such a complex system. In the US, we employed a three-party checks and balances system to try to solve this problem in a more distributed manner.

Obviously there might be unsolved issues around a) preventing such a system from decaying over time and b) providing some sort of mechanism for people born into a governing society they do not agree with to travel to another society without unachievable economic hurdles. This of course too has limits.

I do differ with you in that I do not believe another entity has the right to compel anyone to do anything. The judge can take my phone, but if it's locked, I cannot be compelled to produce a password. This is established by the fifth amendment, but would probably be better formalized today in a new constitution.