I don’t think it’s too hard to figure out. Natural selection disfavors a strategy that requires significant investment in non-reproductive activity (education, career ladder-climbing) during prime reproductive years.

Natural selection works on genes, not individuals. How do my career choices affect my brother's and sister's children?

The selection part works on individuals, what changes from selection is the frequency of alleles in the population. By not having children the alleles you carry will become rarer in the population of the next generation (all things being equal).

Also, having children later in life reduces the role that grandparents play in a childs life, reducing an important safety net.

Not having children removes the grandparents. Hard to be a grandparent unless you have children.

Or removing/reducing neurotic (or even abusive) family nonsense, depending on the family in question.

Not just how many children the groups have, but also how many of those make it to healthy adulthood to reproduce themselves, etc.

"Smart people" have fewer children but they're more likely to be successful.