The agreement to be a part of the society is an infringement on personal freedoms by definition - individuals agrees to constrain their behaviors to a set of rules, in return getting the benefit of scale.

A healthy society by definition aims to maximize the sum of benefit minus the inconveniences of constraints.

In the old days of the village life one could forfeit the impacts to their personal freedom, abandon society and go live in the forest on their own.

Unfortunately the modern societies do not give this option. It would be truly helpful to give some perspective to people.