The heritability of fertility is about 50%, I believe. So about half of the variance in number of offspring is genetic. The remainder is other stuff: culture, chance, etc. I would imagine that the genetic component is multi faceted. Some of it might be a desire to have children. Some of it might be a resistance to certain ideas or lifestyles that result in fewer offspring (if that seems far fetched, consider first that political orientation is also about 50% heritable).