You have indeed captured the standard reasoning. But when a group of people forms an exclusive club, the result is inevitably politics. And politics of a kind that is bad for everyone. It is bad when people in power do it. But it is also bad when people who are disenfranchised do it.

The specific comment in question came about as follows. It was triggered by her recommending a book club to me, then having the organizer ask me not to come "because it would change the dynamic too much to have a man there." This lead to the woman who recommended it to me dropping out of said club because it became something which she did not wish to be associated with.