I think the key mitigation is:

  >  "I do not imply, for instance, that we should always suppress the utterance of intolerant philosophies; as long as we can counter them by rational argument and keep them in check by public opinion, suppression would certainly be unwise. But we should claim the right to suppress them if necessary even by force; for it may easily turn out that they are not prepared to meet us on the level of rational argument, but begin by denouncing all argument"

But I understand your point. To that end, attacking illegal immigrants would be "intolerant". But what about advocating their arrest and deportation?