Banning criminals is a popular rhetorical construction used by those who do not have felony records, in order to scaremonger others into supporting an argument that promotes existing biases and status quos; in this case, in favor of the “sexual misconduct is disregarded” bias.
It's a tough philosophical space balancing between free speech/association and criminal discrimination and I think it shouldn't be surprising that an ecosystem that produced a "free speech absolutist" like Musk who bans dissenting opinions would also create a group of people who think that everyone has to tolerate them no matter how abhorrent their behavior or ideas as long as it doesn't cross a line into "criminal".
> as long as it doesn’t cross a line into “excluding them”.
"Oh no, a problematic white man suffered a consequence. I'm a problematic white dude, I wouldn't want to be kicked out! This should be illegal!"
Alternative language for what you're saying: Should we force people to allow everyone access to them/their company/their spaces? If not, how do we determine what groups are and aren't protected?
Tangential: Why would banning criminals be fine for you, and what crimes warrant exclusion? Should tax cheats be excluded from chess orgs? What about someone with a DUI?