Counting neutral variation (the vast majority of genetic variation). You can get plenty of variation in non-coding DNA from drift without any effect on the variance in a trait. In fact, you rarely actually get effects on highly polygenic traits from drift without selection (imagine random directions chosen independently for each of the relevant alleles, and think in terms of the law of large numbers).
If you're getting this from Oded Galor or Deidre McCloskey, they're generally smart people but don't understand population genetics well.
>Studies on over 700 participants show that individuals with larger brain volumes have higher IQ scores. About two dozen studies using magnetic resonance imaging (MRI) to measure the volume of the human brain have found an overall correlation with IQ of greater than .40
>Within each race, cranial capacity correlated with IQ scores. By age 7, the Asian American children averaged an IQ of 110; the White children, 102; and the Black children 90. Because the Asian American children were the shortest in stature and the lightest in weight while the Black children were the tallest in stature and the heaviest in weight, these average race differences in brain-size/IQ relations were not due to body size.
>Overall, MRI studies show that brain size is related to IQ differences within race. Moreover, the three-way pattern of group differences in average brain size is detectable at birth. https://www1.udel.edu/educ/gottfredson/30years/Rushton-Jense...
I suggested looking and comparing the magnitudes between different factors and putting them into order of importance.
We know that taller people have higher intelligence than shorter people on average [1]. Compare effect sizes between black and whites and tall and short white males. Now ask if the difference between ethnic groups is important or not.
[1] : https://en.wikipedia.org/wiki/Height_and_intelligence
ps. 90% of genetic variance in human races exists in the African continent. If we clean out all environmental factors, the "smart genes" are probably in Africa. We just need to find them and distribute them.