Someone of mediocre intelligence is not harmed by being in a society of people much smarter than him. On the contrary - he benefits greatly because smarter societies are safer, fairer, and take care of citizens (including the dumb ones).
This is revealed by real-life preferences: People from low IQ countries consistently try very hard to migrate to societies of smarter people. People from low IQ regions try to get into neighborhoods and schools of higher IQ people.
It's like complaining that Einstein existed because my grandpa didn't get to invent relativity. Well, my grandpa was never going to invent relativity. The fact that Einstein existed just means that the rest of us now get to enjoy the positive externalities created by his ability.
It's true that if everyone else is smarter, your relative social status would be lower. But this isn't an argument that scales up, since relative social status is in fixed total supply anyway.
Or am I making a mistake by trying to apply logic here?
Mother Nature has no logic. Or rather, if she has logic, it's the logic of simple chromosomal propagation on galactic timescales. She just keeps propagating different combinations and permutations of genes blindly. Then, for instance, if some cataclysmic environmental change happens she ruthlessly culls all of the gene combos that are not able to survive the new environmental reality. Or it could be a new predator. Or a new asteroid strike. Or a new bacteria. Or what have you.
Point is, in the case of our species, she doesn't make people "dumber", or "smarter", or "faster", or "stronger", or "slower", or "darker". She just blindly makes people. Of as many kinds as she possibly can. So that when the change roulette wheel spins, she's already laid bets on every number on the table, black and red alike. Thus giving her the best chance at a win. If I believed in this sort of thing, I might say that Father Time has taught Mother Nature that the shotgun approach is the best survival bet she can make. (And he probably taught that lesson the hard way.)
Contrast that blind propagation strategy against what we would be doing, which is laying odds on certain, but not all, numbers on the hypothetical roulette table. You can kind of see why people might question the wisdom of such a targeted strategy.
Today's genetically improved babies could be sub-optimal by the time they hit kindergarten and just plain old obsolete before they go into high school. And the bad news is, I could see the rate of improvement increasing over time. So that material improvements are being made year to year.
Doesn't bode well for when you're going to look for a job at 29 and you have to compete with 28 year olds. Even worse if you're 29 and you have to compete with 25 year olds. And heaven help you if you're 29 and you have to compete with a 20 year old.