This is actually a good point, and it's important to realize that even Karl Popper acknowledged this. He explicitly argues that some limited tolerance should be extended even to the intolerant to the extent practical, since this helps promote the norm of tolerance in the first place even when intolerant ideas might otherwise appear to be prevalent. The Paradox of Tolerance is more about acknowledging that limits to tolerance must exist somewhere than an actual argument for narrowing those limits.