See, for example, this long thread where this conflation is made: https://news.ycombinator.com/item?id=8368876
The claim is logically false - you can't prove a normative claim without a normative premise. In my example above, one possible normative hypothesis would be "and it's morally wrong to use incorrect premises to make decisions" (or possibly "it's morally wrong to use incorrect racial premises to make decisions").
Now our algorithms are revealing that underlying positive claim (e.g., "racial stereotypes are incorrect") might not be true either.
This is confusing, and rightly so - it demonstrates our entire belief system was predicated on nonsense. Hopefully we'll actually respond with some useful analytical philosophy.
But if a neurologist proves that sexual orientation is acquired during childhood, are we going to throw out gay rights?
In any case, I think these moral arguments are (a) more of a counterpoint argument to the pseudoscientific racism of the Nazis and their predecessors and (b) more about innate genetic potential than socio-cultural traits.
I think the stronger moral argument is about equality of opportunity and (hat tip to a charmingly american ethos) individual merit. The idea is that it is more moral to judge people by their personal merit, rather than profiling and assuming even if that gives you a better chance of being correct.
Any imperfect criterion will always exist at the expense of some group of people (tautologically, the group of people to which the criterion is unfavorable). What difference does it make, ethically, if the group happens to be identifiable?
I agree that this stuff is hard to nail down in a logically consistent way. But, this stuff has a history. Group discrimination has had horrible results in the past. This is our sloppy duct tape fix. Algorithm based selection is now challenging the assumptions.
If an admissions officer offers places only to handsome, white men from upper class backgrounds, we see that as unfair because these things have no direct impact on their value as students. He's an ass. For the sake of argument, imagine an algorithm arrives at the same conclusion as the ass. Our moral instinct is confused. Societal impact is similar.
There certainly is something dehumanizing about selection algorithms making decisions. I hate buying insurance or being assessed for loans. It feels bad.
Basically, I don't think you can reductio ad absurdum this argument away. There are inconsistencies in our moral sentiments.