I think even in those situations, telling them that you never felt like you fit is still the right approach. I there is actual wrongdoing, report them. But if it's a boy's club, it's probably better to phrase it as such than point to each individual actor as sexist.
I agree it's probably best to start from a position that it's just a cultural mismatch. I imagine sometimes it gets entangled with a lot of other issues though, whether rightfully so or in perception only.