I get what you're saying, but I think facts suggest that this isn't downside with significant impact. Populations seem to grow just fine in all sexually reproducing species as long as conditions are ok.

Even some of the poorest countries have very high fertility rates, as high as 6-7 births per female, so availability of women is not an issue.