We aren't talking about how things are, we are talking about how things should be, specifically, whether there should be vaccine mandates or not. Other existing violations of rights are wrong also.

So it is fundamentally about philosophy of governance. You believe society should come first, individuals can be compelled to benefit society even potentially at great cost to themselves, and that a person is morally wrong for not wanting to go along with that. I believe that it is morally wrong to expect a person to value anyone over themselves that they have not chosen to, or to be compelled to do anything on behalf of someone else.