Possibly. But if someone is actually unfit enough to be "gravely" hurt, and multiple generations of their ancestors were all too unproductive to afford IVF and pre-selection even when not under full competitive pressure from superhumans ... then what kind of a person is this, who we are so concerned about? Does his/her loss of competitiveness deprive humanity of so much, that it is better to hold back humanity's best+brightest?

The slippery slope argument usually advanced involves what happens when technology improves and allows bigger single-generation deltas for the existing economic elite.

But, in any case, many people find the idea that the less-genetically-fortunate should suffer to avoid “holding back” genetic superiors to be morally repugnant even when dealing with natural variations, much less when dealing with advantages available only to offspring of pre-existing economic elites.

Ah, in this case we have an isomorphism to the argument between "no child left behind" and private schools.