Figures 1 and 2 of https://www.gwern.net/docs/iq/2007-zagorsky.pdf strongly support the idea that it's the outliers which have the biggest impact on the correlation between iq and income/wealth; specifically, the low-iq outliers.