Same trait may have different effects in different environments. In stone age, bad eyesight might have been a severe evolutionary disadvantage. After we invented eyeglasses, bad eyesight has no effect on your survival or number of children.

In a culture where large families were normal, perhaps genetic traits that push you towards having a large family were hidden by cultural aspects that made you have a large family. Only in an environment where having a large family is not the norm, will they surface and have an effect.

Just speculating here.