It's more 'stop being a prick and be polite'. Society is a series of rules, stated or otherwise, that the generality gravitate around. It's not rocket science, it's just manners.

One has every right not to wear a mask, just as a business owner has every right to deny someone access to their facilities.

Masks are irritating, but the sheer histrionics that they are causing in some quarters is ludicrous and is clearly more 'anti-science' (if that's really your criticism here) than the masked contingent is.