Point of consideration: supremely fit men in history (in a Darwinian sense) could have hundreds or thousands of children. Big wins can have big genetic payoffs. Supposedly Genghis Khan has millions of living descendants today, with perhaps as many as 1 in 200 men being direct descendants [1]. Supremely fit women can only have a limited number of children. This difference likely has some kind of effect on reproductive strategies, and what kinds of variation are selected for.

[1] https://www.discovermagazine.com/the-sciences/1-in-200-men-d...