The same may be true here, and I think it's the most likely explanation.

I'd be interested in whether the same model can be trained to predict patient wealth, hair color, style of clothing, religion, etc. from the same x-ray data sets.

While "faster than light neutrino" was highly unexpected and rather suspect from the start, the "bone geometry differs slightly between ethnic groups" is well established among the anthropologists of humans. There are also parallels in wider biology of animals - mentioning that to underscore it's as scientifically expected, and not merely construed for humans alone.

The question here was how exactly is AI detecting it this well from chest X-rays; the question centered around AI and possibly if it would unexpectedly influence the medical processes - rather than around the bone geometry itself.

For sake of example, a random link from google search: https://www.researchgate.net/publication/24427702_Ethnic_dif...

This specific model's ability to do it from a 64 pixel version of said x-ray makes me skeptical it's doing so successfully.

That's actually a great example of this problem, though.

https://www.theverge.com/21298762/face-depixelizer-ai-machin...

> It’s a startling image that illustrates the deep-rooted biases of AI research. Input a low-resolution picture of Barack Obama, the first black president of the United States, into an algorithm designed to generate depixelated faces, and the output is a white man.

> It’s not just Obama, either. Get the same algorithm to generate high-resolution images of actress Lucy Liu or congresswoman Alexandria Ocasio-Cortez from low-resolution inputs, and the resulting faces look distinctly white. As one popular tweet quoting the Obama example put it: “This image speaks volumes about the dangers of bias in AI.”

The notion that a classifier can reliably identify race based on an 8x8 grayscale is risible.

The same is... not remotely true for humans, or even two chest x-rays of the same human.

Seeing as this would be easy to do, I imagine that if it is at all plausible from what they know that it is getting information from anything other than the x-ray scan, that they would have already tried this?

I do wonder how good of a predictor something would be if it just went off the average brightness of the image. Probably very bad, but maybe better than chance? Well, better than chance on the training set is to be expected, the question I guess is whether it would be better than chance on the test or validation set (I’m not confident in my understanding of the distinction between testing set and validation set. Is the idea that if you are using the score on the testing set to decide when to stop training, and maybe what hyper parameters to use or something, and other things to determine which model, you only try the model on the validation set once you have decided on your final version of the model?)

It's confusing, not least because people refer to "testing" when they mean "validation".

So, suppose you have a dataset, let's call it D, and it doesn't matter what's in it other than "instances". To train a classifier you start by creating two partitions of D: a trainign partition (the "training set"), and a testing partition (the "testing set"). We'll denote them by T₁ for the training set and T₂ for the testing set.

It's typical to use most of D as a training set, for example you may choose 80% of D to be T₁ and 20% to be T₂. Obviously T₁ ∩ T₂ = ∅ and T₁ ∪ T₂ = D.

Now, because T₁ is four times the size of T₂ it's very likely that when you test your classifer on T₂, it will appear much better than it is, just because most of the instances in T₁ aren't represented (by similar instances) in T₂. This is called overfitting to the training set. One way to mitigate it is to perform cross-validation, the most common type of which is k-fold cross-validation.

In k-fold cross-validation, you further partition T₁ to k partitions, or "folds", and then hold out each i'th partition, for i ∈ [1,k], use all the rest k-1 partitions as a training set and test on the i'th held-out partition _during training_. So you train your classifier on partitions 1 ... k minus i, test it on partition i, and repeat this process for all i, recording the performance (accuracy, F1, ROC etc, whatever your metric is). Then you choose the model that performed the best on your chosen metric.

And then you test it on T₂.

To avoid confusion between the k folds of T₁ that you use for testing your training models during cross-validation, on the one hand, and T₂, that you use for testing the model that performed best on cross-validation, on the other hand, we call the testing process performed on the k folds "validation" and each i'th subset of T₁ used for validation a "validation set". And we just call T₂ the "testing set".

The confusion arises because we do actually _test_ on sub-sets of T₁. But T₂ is always the "testing set" and it's never "seen" during training.

As to hyperparameter tuning, this is done _on the testing set_, i.e. T₂. This is A Very Bad Thing™ but there you go. Once you train a classifier and find out that it sucks on T₂, what do you do? Well, you tune the classifier's hyperparameters. Or do a grid search to automate the process. So eventually you overfit your classifier to the test set, because you now essentially have no "unseen" data instances in T₂ - the classifier didn't see the instances in T₂ during training but the trainer did, or, worse, the grid search did, and the classifier's hyperparameters were tuned according to that knowledge. How to avoid that, is a big question, but anyway that's what is done in practice, and the reason for that is that when you do Big Data, you end up needing so much data that despite having terrabytes of it, you never have enough.

That... that doesn't influence one of the presumed ways the NN categorizes images: the trend in bone geometry. The "blobs", while fuzzy, still largely retain the relative proportions to each other. Or, in other words, proportions of image elements are invariant for operations of scaling and of blurring.