The position that Twitter can’t ban people is the anti-speech position.

Only one side of the argument wants people to say anything they want, once you realize association is a form of speech. The side that supports Twitter’s expression of speech is closer to “everyone gets a say” than the side forcing Twitter to publish content it doesn’t want to.

You still get to speak, just not on Twitter. You don’t get to trample on Twitter’s rights. That’s anti-speech.