The difficulty is that the aberrations in a large group can have disproportionate impact on those outside of the group.

So, men who abuse women may be the exception, but for those abused women, their perception and comfort with men may be permanently changed by those exceptions. Randall is not saying all men are the exceptions. Rather, he is saying recognize that those exceptions have had an enormous impact on some women's lives, including simple things like being uncomfortable publicly identifying their gender.