A casual encounter with a SARS-CoV-2 infected individual coughing on you without a mask can easily land someone in the ICU.
Being incapacitated by drugs and alcohol are permissible in certain situations (being in your own home), not in others (driving on the freeway).
In a similar fashion, not wearing a mask is permissible in some situations (being in your own home), not in others (standing six inches behind me in line at the grocery store).
which is why all those things are limited by law. I can't smoke in public spaces, I can't drive drunk, etc.
They are not 'rights' as you seem to argue, and they are limited/regulated just like a million other behaviors. Wearing a mask is no different. Personally, I'd say mask regulations are the equivalent of seatbelt laws - maybe uncomfortable, or inconvenient but basically harmless.