There's an old saying that comes to mind "in theory both theory and practice are the same, in practice they are not." The fact of the matter is anti-maskers and anti-vaxxers both exist and both endanger the health of those around them. This issue pits individual rights against community needs and as such it often goes awry. There are those who will make small personal sacrifices for the communal good and there are those who won't. The problem is communities are comprised of both kinds of people. That's why this notion you've expressed doesn't work as well in practice as it seems it should in theory.