If you are implying that liberty is zero sum then I disagree.
I'm also uninterested in average liberty. Individual liberty is of utmost importance. Individual rights extend as far as infringing on the rights of others. This is important only in the context of preserving further individual liberty.
(Case 1) A world in which individuals are completely free to do with their wealth as they will (so, no taxes, regulations, etc).
(Case 2) A world in which individuals are absolutely prohibited from wealth transfer. At the time of their death, all individual assets / privileges are seized, then redistributed evenly to the next generation.
Select a random individual from a new generation in each case.
Would you disagree that they have different effective individual liberty? That is, the options available to them to pursue in life?
Essentially, I see effective liberty as the product of rights and means to exercise those rights. And I believe the greatest individual liberty cannot be created without balancing both requirements.
What you're advocating is mob rule and that's why it is scary.
Obviously there are different standards of proof but the presumption is always innocence and there is always a standard beyond how it "looks". This is preferable because it maximizes liberty.