We have made up a huge number of arbitrary rules, some at a very broad scale (for instance, that you must shoes and shirts to go into most businesses), others idiosyncratic and basically between one person and another (no cats in this apartment). They all restrict our choices and our freedom and our actions.
We have also set specific limits on what sort of rules we can make up. You can say "no cats" in an apartment, fine, but you can't say "no pregnant women". You can fire someone for showing up at work one day with a face tattoo but not for changing their religion.
"Not wanting to get a vaccine" or "not wanting to wear a face mask" is not currently a recognized protected class (although not being able to because of a medical condition probably is, but hey, of the many many things I am not, I am not a lawyer). There's nothing saying it couldn't be -- we didn't used to have a concept of "protected class", we just had to make it up at some point -- but that's one way to frame this discussion.
The power to exclude is the power to coerce.
Society is coercing people into taking the vaccine by requiring it for any activities outside the home. Want to go to the grocery store? Get a vaccine or wear a mask. Want to go to Disneyland? Get a vaccine or wear a mask. Want to attend a conference? Get a vaccine or take a test and wear a mask. The mask and testing mandates are an effort to wear down anti-vaccine people instead of letting them be.
Compare: "You can murder someone, but then freedoms will be taken from you, because you could hurt someone with these freedoms. You won't get brainwashed out of it, but if you do commit murder, your freedoms are taken away."
Not arguing against the policy, but it in practice, it _is_ the same.