Probably, right. The cave men or before that probably only had one or a few chances to mate while in our society we live in couples and can copulate many times per month, year in and year out, and thus a lower sperm count is countered by an increase in the number of tries.

I also find it hard to believe that sperm count should be such an important factor, there is still such a huge amount or sperms that have a chance to reach the egg.