Credible or actual harm is useable as a legal standard. If you wanted to outlaw "hate speech" what standard could be used to do that? How could you prevent abuse of those laws in order to suppress unpopular or otherwise subversive speech?

In other words, it might sound like a good idea, but there's absolutely no way to prevent a legal standard that does not stem from actual or credible harm from being abused by the powerful to suppress the powerless.