You just need different relative reproduction rates or different levels of assortative mating. For example, the Khmer Rouge may have affected Cambodia's IQ distribution (negatively) but Cambodians don't now have "biological differences" per se.