The point is “the AI might not be doing what you think it is doing, because it doesn’t actually understand the goal”.

You're misunderstanding the context of the thread. At this point in the thread, we're talking about whether or not there is any biological basis for race, not about the AI example in particular. The premise here is that there is a biological basis--that race is a label applied to a cluster of genetic traits (various facial shape features, skin color, etc) which are almost certainly genetically constructed.

The comment I replied to stated "clearly there must be some biological basis for race, if an AI can determine race from an x-ray".

I'm pointing out that's not necessarily true. First, we have to prove that the AI is determining race from an x-ray. It's effectiveness at doing it via 64 pixels makes me skeptical that it is.

After that, we get into sticky territory determining whether any biological differences we can identify between races are caused by genetics, or environmental/societal factors like disparities in healthcare, diet, neighborhoods built over old SuperFund sites (https://en.wikipedia.org/wiki/Love_Canal), etc.

Ah, my mistake.