The debate reminds me of the debate about smoking.
For a long time, smoking "probably" caused cancer. Meaning, almost certainly. There was plenty of evidence.
But for a long time, cigarette companies said it wasn't "proven" that it caused cancer. Meaning, let's carry on selling, advertising, consuming it in large quantities.
It would take a few more decades to reach the standard of "proof".
So smoking continued, people died.
Rather than following rational life-optimising behaviour based on the growing body of data.
As you illustrated in your example, the medical establishment is often disastrously wrong, so why shouldn't we take their recommendations with a grain of salt?
They have to take a little care where they swing their arms.
In fact some people don't take that much care, and we usually say it is the people who physically dominate a space by being inconsiderate of others who are acting entitled and privileged.
So no, I don't think it's entitled or privileged to ask people to reduce significant[+] aggregate harm to others, at minor inconvenience to themselves.
However, I think it's entitled and privileged for people to decide that they don't have to wear a mask based on their "personal risk assessment for themselves", on the theory that other people can just quit their jobs or whatever and stay locked up at home to feel safe from the unmasked crowds outside. That's the "if people want to feel safe they can stay at home if they want" theory. That's not a reasonable imposition, and does not constitute the reasonable balance of needs that we'd call a free society.
For those for whom it's particularly uncomfortable, I don't know the policy elsewhere, but in the UK, it's permitted to not wear a mask if you have a reason for not wearing one. Reasons are quite reasonable: For example if you're with someone who lip reads, or they give you anxiety or whatever.
The goal is to ensure that almost all people wear one, not that every last person must wear one if they have a reasonable reason not to.
The reason for enforced policy is because too many people won't wear one at all without a stronger incentive. Just asking people to not harm others isn't enough, it seems. (And people really can't get their intuitions around exponential growth and clustering, so they mis-estimate aggregate risks to others by a lot, even when they have numbers to go on.)
[+] (It affects the R number. The difference between R > 1 and R < 1 is pandemic versus disease dying out. It's also the difference between a long pandemic with onerous economic and social consequences, and something that is manageable. The overall effect is profoundly significant in aggregate.)
The problem with that framing is that most people quickly drop the "asymptomatic" bit, and so now we have people with symptoms going shopping wearing a mask. That mask isn't going to do much to protect other people, especially when you consider it's likely to be 2 layers of cotton, poorly fitted, with poor hand hygiene.