Regardless, call me a fanatic, but a big part of individual rights, and I believe liberal democracy, not just libertarianism, is not to be subject to other people's panic. Whether that is a border wall or its enforced mobility and gathering restrictions, there are things the government should just not be allowed to do, because they allow for a tyranny of the majority.
There have been discussions, long before covid and largely precipitated by Trump, about how our system only works if we all generally follow some norms and don't abandon them in some kind of populist panic. I would argue this is what's happened here.
It's not acceptable that there are magic words (science says, "public health is a real thing that matters" as you say) that anyone can say that allows government to bypass civil liberties and impose stuff on people. Freedoms only actually exist if there is weight behind them and they can't just be discarded when someone shouts that lives are at stake.
For me it's not a question of disagreeing with any science about how covid is transmitted or vaccines or severity. People should behave sensibly about it and do what they can to limit the impact. Giving power to government just creates populist measures designed to cater to the masses and make them feel good (exactly like a border wall). This is not a power government should have. Call me a fanatic (religious type as you say)
If you say instead that it is a legit function of government, but shouldn't be used in this case because on balance the risk is too small to sacrifice liberty for, that's certainly a discussion that could be had.
As a libertarian I wholeheartedly reject this fig leaf of "freedom" touted by anti-maskers. Freedom is an ability to choose the right thing, not simply being a contrarian for contrarianism's sake. I think hard drugs should be legal, but I similarly wouldn't support people becoming junkies to spite society.