It's essentially the paradox of tolerance. The solution to that paradox is to ignore it. To quote Chidi Anagonye, this is why people hate moral philosophy professors.