When Popper described intolerance, he spoke of people using 'fists or pistols' to shut down the tolerant and he predicated the very idea on the right to self defense.
The right of self defense is and long has been defense against immanent physical harm, for the one who does not start the fight.
That said, the OP is also not right to suggest that intolerance is simple to identify. Like the other comments here say, it's subjective.
Then again, most things in life are subjective. Society has to be pragmatic sometimes and ignore that. Otherwise everything, eg: the legal system, falls apart.