Quite a number of scientists are studying how brain development is affected by very early experiences, and so far it seems like it might be more than most people thought. Basic experiences of childhood like breastfeeding, number of words heard per day, amount of music heard, even the number of books in a house, have significant correlations with later metrics of mental success like IQ, grades, career outcomes, etc.

By the time a person is old enough to be taking math classes, their "innate" abilities are a combination of nature and nurture. Hitting a "breakpoint" might be the result of deficiencies in their nurture, rather than truly innate (i.e. genetically determined).

The question is, if you took every person and raised them in the optimal way, how far could they go in math, programming, or any other pursuit? I think it's obvious that there would still be differentiation according to talent, but I also think it's likely that the average would be a lot higher than it is.