You may or may not have noticed that extreme poverty and hardship doesn't lead to less offspring, in fact the opposite occurs. Look at birth rates between wealthy and poor countries or groups.