Because compared with bench-pressing, claims of mathematically ability being better in men (and partially ordered, to boot) is seriously jumping the gun.
We know what's involved in a bench press. We understand how testosterone stimulates the production of muscle. We are nowhere close with mathematical ability. We have no theory of mathematically ability -- we really don't know what it means, or if the simplest metrics are even useful for higher level math. We have no experimental results, because we have no controlled variables. We have few pieces of data, none of which are conclusively disentangled from cultural and historical influence.
In the past two decades, the number of women scoring highly on the IMO, the IOI, the Putnam, and SMPY has gone up by roughly a factor of six. Doubtful that the number of girls with 'math talent genes' have sextupled that quickly. Isn't this evidence that we should hold off on our conclusions?
But we need to be open to the fact that they are there. It hinders scientific and social progress to scream sexism any time someone suggests the sexes might be different in some way.
It might be absurd to assert equality, but it's a decent postulate to take while we don't know for sure. And I think asserting the certainty of particular innate differences, without sufficient evidence, is more dubious than the corresponding assertion of equality.
Darwinian theories about women being wired differently than men are perpetuated because science backs them up. We don't know exactly how, or how much of the difference is genetic vs. cultural, but it's just a reality that men and women have different motivations, aptitudes, etc.
Nobody would argue that evolution has made us different physically. Men are much stronger and better coordinated. Women are more able to bear pain and often have stronger senses of smell and taste. Why would it not stand to reason that evolution might make each sex better at some mental tasks than the other, just as it has physical? It's not sexist to point out that male weight-lifters are able to bench more because men have been designed by evolution to be stronger. Why is it sexist to say we show up more frequently in science departments because we have also been designed by evolution to be better at math?
Sexism is not believing that one sex is different than another, it's believing that those differences make it inferior.