It's a balancing act. We limit <some thing> if the damage to society caused by it is greater than <some amount>. With the caveat that getting those limits put in place can be complicated, given the influence of money on political will.

- We limit drinking by minors

- We limit smoking to outdoor areas

- We limit driving over certain speeds

We limit many, many things that an individual can do based on how they impact society as a whole. We even limit some things an individual can do that have no impact on society (but I disagree with this completely).

That is how societies work.