Just because you can apply a rule with one set of nouns and achieve an unjust outcome doesn't mean the rule is bad.

There's no daylight between "we can't restrict shitty speech because one day we might restrict non-shitty speech" and "we can't have laws against bad things because one day we might have laws against non-bad things". It's an obviously stupid argument and it's not any less stupid because the operating verb is speech.