I feel like that stigma would, to a large degree, be associated with the identity of "outsider" that you'd acquire in leaving the original group. If that group didn't want you, why would we?