I'd argue this is more about defaults. Speech defaults to free and is well protected with extremely rare/narrow exceptions, violence defaults to controlled.

Some forms of violence of private citizens are not illegal (self-defense), but I think we're better off in a society where violence is default illegal.

Speech and violence are pretty different things.

But violence is just physical interaction that we define as harmful by intent - a hug might be well be painful to someone but you can't be prosecuted for it. A solid punch in the arm might cause no measurable harm at all to particular individuals but it's still considered assault. Yet if you intentionally cause permanent mental health damage to another through repeated intentionally offensive and/or disparaging speech, why should you should get a free pass?