Think of it from the individual-up. Even if the half of people with affected IQs were spread proportionally throughout the IQ bell curve, they'd have lower IQs than they did before. The other half would raise just as much as the affected half lowered, but the affected half still have lower IQs.