I think this is an excellent example of what I meant about "how direct the harm is" mattering.
Imagine, if you will, a continuum ranging from absolutely direct harm to absolutely indirect harm.
On the absolute left, imagine person A stabbing person B. That's about as direct as you can get, right? And I also think there's not a ton of people who would disagree that stabbing someone should be banned (let's ignore self defense arguments).
On the absolute right, imagine person A moves a chair. Someone trips over this chair. That person ends up in a foul mood and cuts someone off in traffic. This causes a traffic jam. In this traffic jam, a refrigerated truck breaks down and a bunch of food is ruined. More food is ordered, which takes up space in a shipping container that would have gone to medicine, had the food not been ordered. Person B dies because of a lack of medicine that would have been there had person A not moved the chair.
This was about as indirect of series of events as I could come up with, but I think you get my point here. There's a direct causal chain between person A's actions and person B's death in both scenarios, but I can't imagine very many people really condemning person A in the second one. In fact I think someone trying to ban moving chairs because of this incident would be soundly ridiculed.
So where does going outside without a mask during a pandemic sit on this continuum? Well, going outside without a mask, if you have a high viral load, and someone comes close to you, there's definitely a chance they could die. That's definitely not nearly as direct of harm as stabbing someone. It's also clearly not nearly as indirect as that tortured example I came up with in the second one.
Personal opinion? It's definitely farther to the left than the right. Say, somewhere between 1/2 and 3/4 of the way to the left. Maybe somewhere around 2/3 of the way there.
Someone's opinion on something such as a mask mandate, is going to depend on if they view that as direct enough harm for the government to have a legitimate interest in regulating.
A clear historical example is America's conflict around slavery. The primary southern justification for creating the Confederacy was freedom. That continued after the Civil War; see the UDC catechisms for example. "States' rights" they'll shout, meaning the right of the white people in power to treat black people like livestock and keeping that in that condition through systematic violent abuse. And you'll still find people today arguing the point.
A modern example is public health. Everything from preventing toxic waste dumping to COVID mask mandates is met with cries of "FREEDOM!" Right now there is a large constituency in the US for the right to do whatever they please even if that means they will kill people by giving them a disease.
I agree that what constitutes "harm" can be a tricky question for some issues. But I think we can agree that death qualifies. And that people who have never left the house unclothed having to wear another half-ounce of fabric in public is not.