I think the line has to be physical violence, else we'd allow very few people to speak as any other interpretation kind of dismisses just about anything and everything:
Abortion rights
Abortion opponents
Trade agreements
etc.
> Is yelling abuse at people a violent act
No, it's an abusive act though.
The simple answer to your question is: The definition of violence by the courts is physical.
And it's probably a good decision. Otherwise the rabbit hole will take us bad places.
This is not an easy problem, and 'open calls to violence' are not a trivial solution.
I also fear universities which seek to over-represent minority points for the sake of balance, in the same way that news programs often do.