Yeah, abuse is not speech. There's a difference between voicing your opinion and harassing someone. What's more, harassing someone is often done with the intent to silence them. And even if it's not the intent, it's often the result. Harassers are not defending free speech, they're attacking it.
And even apart from that, I think everybody has a right to choose who they associate with, and that does not have to include harassers. That's more relevant to social media and less to teaching classes of course, which do need to be open to everybody. But if you're choosing to follow a class with the specific purpose to harass someone, there's something deeply wrong with you.
People are weird.