In the same way when you sit down to play a boardgame with someone and they draw extra cards, fudge die rolls, and flip over the table when a turn goes against them: they're no longer playing the agreed-on game at that point, and you and the other players don't have to accept their behavior as part of the play.

There is a baseline set of rules (in the USA we have the Constitution) for how the game works. Like a person's freedom to swing their arm, a voter's freedom on what to vote for must stop at the limit of the other voters' rights. The "Tyranny of the Majority" is a related concept as well.