The issue is the harmful actions and the people who commit them. That's why there's an exception for speech that incites violence.

Outside of that, rhetoric that you consider hateful is not a direct cause of any harm. That is a dangerous road to go down and it's far better to counter with your own speech instead.