This is a good point. I guess the difference here will be "how much do we trust human judgement?" and "how much do we trust AI predictions?". If AIs make 10% less errors than humans but we have 50% more trust in them, that's a problem. Eugenics were not a problem because science was wrong, but because people blindly trusted it to apply where it should not.

There are a lot of things to consider, here, those are exciting times for thinking.