This seems extremely unlikely. If the server admin doesn't like you enough to ban you, I strongly doubt you're going to have many followers there.
In any case, however likely or unlikely it is in practice, the incentives here seem exactly right to me. If you want the benefits of being part of a community, you have to respect its norms in your interactions with others in the community. The possibility of losing connections with people in that community if you are forced to leave it is part of the deal.
And I think that's great.
I had over 10 thousand and had many interactions with people but wouldn't feel like bothering them on other platforms (unless it was of the same nature).
And the few people who bothered to contact me personally felt a little awkward too. I don't necessarily want to be talking to these people directly.
The magic is you just publish whatever you want and people can look at it. They can comment but there's no obligation to respond. Having actual conversations with that many people would be a huge chore. The whole thing isn't about having relationships with people at all but about spreading information. It could be about yourself in which case there would be some implicit one way relationship but that's just one use case.
People are thinking of others as consumers while expecting to be treated as a friend?
But yes, I guess you can start over somewhere else while on big ones you're out forever. So in that sense I guess it is a tiny improvement. But if most of your followers were concentrated on server[s] that don't like you, you may not be able to reach them anymore.