That something is single sex makes it private, yet also less directly exclusive or discriminatory. It's not public, it's a fraternity with membership, which makes it a privilege one earns. So there's a bar. Privacy provides some relief from impertinent people. If one is ideologically opposed to such gatherings, there isn't much to persuade them otherwise, but I suspect having somewhere to go that is free of them is its own reward.