On the contrary, not having enough diversity can and does put off those who would otherwise make great contributors.
As such, I would oppose any attempts to establish quotas or even aspirational targets for diversity; as you say, forced diversity has negative consequences -- not just for the organization, but also for the individuals involved.
But as much as we shouldn't be trying to bring in individuals who aren't really interested, we need to make sure we're not keeping out individuals who are interested... in a sense, it's more important to look diverse than it is to be diverse.
For example, there was a study done on violent men in prisons and testosterone. The theory that was tested was if testosterone levels were indicator of more violence, for which the data revealed was not the case. Further studies showed that so long testosterone is within natural occurring levels, aggression was not effect by testosterone.
In a wider context, yes, it's a bit more complicated, but Free Software? As a movement, there's power in numbers, and looking to increase that number via a focussed effort on minority groups is simply a smart move.
There's strong evidence that genetic differences account for some behavioural differences between men and women. The ones you suggest do make it clear why you'd expect a skewed distribution of genders in fields such as the army or midwifery. It's much less clear that genetic differences should lead to such an underrepresentation of women in tech fields; in this case I suspect that culture and social inertia are a large part of the explanation.
First on sex/gender: I feel that hormonal differences create psychological differences in the norms between genders. This is why men are generally more aggressive and violent(see higher incarceration rates) and women are more nurturing. These differences lead to people in general choosing careers that reflect their psychology, and forced diversity hurts those that do best in their field.
Then on ethnicity, I feel many of the differences come from both differences in opportunity(which should be overcome) and differences in cultural and social norms, which lead to people choosing different careers.
In my opinion, forced diversity hurts those who would normally gravitate towards those fields because they are displaced by someone who doesn't necessarily want to be there, but just sees opportunity.
Again, I'm not looking to fight. I'd love you opinion on why I'm right or wrong.