Just as important as freedom of speech is freedom of association. If your speech is seen as extremely odious, other people have the right to cut ties with you. And if you're in a prominent officer position, the organization as a whole has a right to disassociate itself from you because of those views.

So is Stallman's speech in question here odious? At first glance, not really. But when you put it in context:

The entire brouhaha starts with a sexual predator whose actions are known to but ignored by associates because money (or equivalently, power). Stallman is defending an associate of his who is tainted by association with Epstein. In the kinder light, this defense is essentially pedantry (just rape, not sexual assault). In the harsher light, this defense is "there's nothing inherently wrong with the entire situation here."

In this situation, I think the harsher light is closer to RMS's intent. He has made statements in the past saying that he believes there is nothing inherently wrong with having sex with children (although he has now walked back those statements). In addition, he appears to have acquired a reputation as a sexual harasser among women at MIT over the past few decades.

With that context, it does look like Stallman shares a lot of attributes with "prominent people whose sexual harassment has been ignored because they're powerful people, and who is not sorry about it." And pushing him out of leadership positions because of those views is acceptable consequences in my opinion. More so because it appears to me that he doesn't appreciate how power differential may affect the ability of people to give consent, he is in a position of power, and he appears to desire consent that may be unwilling.