This happens with any minority group. Race, sex, etc.

The irony is that being around other like individuals brings people together, and people naturally like that. The issue isn't that you're joining up with other similar people, but with keeping dissimilar people out.