Indeed, losing email would be worse, but the usual arguments to let Twitter ban whoever they want don't rely on the premise that getting banned from Twitter isn't that bad.

They absolutely do depend on the premise that getting banned from Twitter isn’t that bad. I’ve made this exact argument here many times before; Twitter doesn’t rise to the level of ubiquity and necessity for us to consider classifying it as a common carrier. Meanwhile I absolutely can see some argument for email getting such a classification.

Any time that we discuss whether or not the government should step in and curtail the actions of a private party, we are explicitly or implicitly balancing the harm of government interference against the harm that such a private action might cause.