It's a good catch to realize that the main downside to this approach is an inevitable lack of diversity in your friend group. That feeling you are basing it on is a general sense of cultural similarity and worldview similarity. Some of those similarities you want but many you don't.