When Popper described intolerance, he spoke of people using 'fists or pistols' to shut down the tolerant and he predicated the very idea on the right to self defense.

The right of self defense is and long has been defense against immanent physical harm, for the one who does not start the fight.