Really? Could you give a reference to anyone ever actually making this argument?
In particular, the controversy surrounding Damore's "manifesto" seems very hard to understand if most people believe that there are differences between the genders that could affect the "outcome"/gender gap in tech/whatever. If this in fact what everyone thinks, then why are so many people upset that someone wrote a document saying this is the case? The whole thing should to be a no-issue? (I guess some of the points of Aaronson's original blog post might be applicable here. But I'm not sure that really improves the situation...)
Secondly, if everyone agrees that there are differences between the genders that could affect for example the gender gap in tech, I have a very hard time understanding how one can routinely blame more or less any difference between the genders on discrimination or some such.
You're changing your original summary of the argument by adding the qualification "that could affect the outcome". The whole question at issue is whether the differences that there are between men and women would be expected to have a significant effect on outcomes in tech.
https://www.youtube.com/watch?v=kasiov0ytEc
Not a really exotic belief in the core social justice circles.
Though advocates tend to avoid stating it outright, since it's so easy to attack and so ridiculous. So it's generally left implicit. E.g. They'll just say, "X field is only 30% women, we must fight the sexism that causes this", leaving it as an unstated assumption.
Firstly, even if there was only this claim and accusation, one might want to defend oneself against it. Especially since this is an almost universally agreed upon "truth" in the western world.
Secondly, motivated by this (perceived) discrimination of women, we as a society have allowed various forms of discrimination of men intended to offset the discrimination of women. This gives an even stronger reason to contest this "truth" if one feels that it is in fact not true.
So what kind of defence is someone who does not agree with the above "truth" allowed? Like how insanely watered down should the defence be in order that one does not "accusing [ones] colleagues of having inferior genes"?