Well, you would be wrong. Because this AI predicts self-reported race, not ancestry.

Think about what this means People who have, say, three black great-grandparents and five white ones, will vary widely in how black they look, and this likely affects how they identify. But there are also invisible markers of ancestry - of the sort you would expect maybe were visible only on an X-ray if you looked carefully. Those could easily break the other way. You could easily get someone who looked white (and thus, probably identified as white), but were "black on the inside", in terms of their less visible characteristics.

Yet the algorithm sees through all that, and manages to see what you feel like? Correctly classifying Shaun King as black and Tom Jones as white? From a 8x8 X-ray picture?

The people who insist that "race is real" should be the most confused by these results, since we know how fuzzily identity is coupled with ancestry, especially in the main groups studied here, black and white Americans.

I'm much more prepared to believe, for instance, that there is something catastrophically bad going on in medical image dataset collection, than I am that self-ID race is nebulously predictable from almost nothing.

You're really missing something obvious here. I don't know if it's the statistical nature of ML or the statistical nature of correlation or what. But the fact that everyone else can see how it probably works while you can't should be a red flag that there's a hole in your understanding somewhere.

Or maybe it's you who should try training a couple of image classification models. I'm sure I've trained far more than most people who are blazé about this.

Why don't you answer the argument instead of trying to convince me it's risky to disagree with the crowd?

The obvious answer is that self-reported race is correlated with many biological traits throughout the body. This is already known. It's no surprise that ML can detect those differences. If somebody's self-identification is abnormal (doesn't know their ancestry, say), then sure, the AI might see through it. Conventional measurements could see through it too.