Even assuming that genetics have absolutely no role in success (a bit unlikely I think), culture is a real thing. Plenty of people will admit they had a huge advantage because their family discussed finance at the dinner table when they grew up, or growing up the child of a business owner.

It's hard to imagine there being no negative side effects to the successful portion of the population shrinking at a faster rate.