It's because it was ostracized for so long. If most of society is telling you that something is wrong with you, then you'll seek out other people who share that trait and develop a sense of community with them. It's normal to identify with your community.