The fact that it works on an 8x8 massively pixelated version of the x-ray points to the possibility that it's not actually working, which would be bad if you based patient treatment decisions on an training set that was actually teaching the AI something else entirely.
What do you mean, not working? That the AI was randomly choosing the correct race 82% of the time by luck?
I'm confused by what your implying because it would seem to me that the authors went through many steps to try to pinpoint how the AI was doing this identification and how baffling it was to everyone that even with a lot of x-ray information removed (8x8 pixels compared to say 4k), it somehow was still correctly picking the race.
What would this "something else entirely" that you are implying actually be?
No; as with the article I linked elsewhere in the thread (https://techcrunch.com/2018/12/31/this-clever-ai-hid-data-fr...), that the AI might have found some other indicator, like filenames in the data set, or metadata in the images that included patient name, or differences in the length of patient name (often redacted by black rectangles in x-rays in training data), or any number of other factors.
This happens all the time in science. As another recent example of "whoops, turned out we were measuring the wrong thing", https://en.wikipedia.org/wiki/Faster-than-light_neutrino_ano...
Another example around AI: https://www.vox.com/recode/2019/12/12/20993665/artificial-in...
> One such résumé-screening tool identified being named Jared and having played lacrosse in high school as the best predictors of job performance, as Quartz reported.
Are lacrosse players naturally better workers? Probably not. Are they probably whiter, wealthier, better networks, etc. than the average population? Probably. These sorts of things - as with the 8x8 pixel example - start to point to confounding variables that need to be worked out and accounted for.
The paper quite explicitly goes into testing and disseminating what exactly the AI detects. Two observations:
- the classification clearly was primarily based on the visual content rather than spurious metadata, because various transformations of the visual content had the expected impact on classification correctness
- the classification clearly wasn't based on one specific feature of the visual content but rather on multiple factors in the visuals, because various transformations to features (including masking out specific features like bone density) produced results matching expectations (usually gradual decrease in accuracy, with some thresholds).
Conversely, if the classification was primarily based on factors other than the visual content, the visual transformations would have had negligible effect - possibly up to a threshold, and then would throw the AI completely off.
The same may be true here, and I think it's the most likely explanation.
I'd be interested in whether the same model can be trained to predict patient wealth, hair color, style of clothing, religion, etc. from the same x-ray data sets.
While "faster than light neutrino" was highly unexpected and rather suspect from the start, the "bone geometry differs slightly between ethnic groups" is well established among the anthropologists of humans. There are also parallels in wider biology of animals - mentioning that to underscore it's as scientifically expected, and not merely construed for humans alone.
The question here was how exactly is AI detecting it this well from chest X-rays; the question centered around AI and possibly if it would unexpectedly influence the medical processes - rather than around the bone geometry itself.
For sake of example, a random link from google search: https://www.researchgate.net/publication/24427702_Ethnic_dif...
This specific model's ability to do it from a 64 pixel version of said x-ray makes me skeptical it's doing so successfully.
That's actually a great example of this problem, though.
https://www.theverge.com/21298762/face-depixelizer-ai-machin...
> It’s a startling image that illustrates the deep-rooted biases of AI research. Input a low-resolution picture of Barack Obama, the first black president of the United States, into an algorithm designed to generate depixelated faces, and the output is a white man.
> It’s not just Obama, either. Get the same algorithm to generate high-resolution images of actress Lucy Liu or congresswoman Alexandria Ocasio-Cortez from low-resolution inputs, and the resulting faces look distinctly white. As one popular tweet quoting the Obama example put it: “This image speaks volumes about the dangers of bias in AI.”
The notion that a classifier can reliably identify race based on an 8x8 grayscale is risible.
The same is... not remotely true for humans, or even two chest x-rays of the same human.
Seeing as this would be easy to do, I imagine that if it is at all plausible from what they know that it is getting information from anything other than the x-ray scan, that they would have already tried this?
I do wonder how good of a predictor something would be if it just went off the average brightness of the image. Probably very bad, but maybe better than chance? Well, better than chance on the training set is to be expected, the question I guess is whether it would be better than chance on the test or validation set (I’m not confident in my understanding of the distinction between testing set and validation set. Is the idea that if you are using the score on the testing set to decide when to stop training, and maybe what hyper parameters to use or something, and other things to determine which model, you only try the model on the validation set once you have decided on your final version of the model?)
It's confusing, not least because people refer to "testing" when they mean "validation".
So, suppose you have a dataset, let's call it D, and it doesn't matter what's in it other than "instances". To train a classifier you start by creating two partitions of D: a trainign partition (the "training set"), and a testing partition (the "testing set"). We'll denote them by T₁ for the training set and T₂ for the testing set.
It's typical to use most of D as a training set, for example you may choose 80% of D to be T₁ and 20% to be T₂. Obviously T₁ ∩ T₂ = ∅ and T₁ ∪ T₂ = D.
Now, because T₁ is four times the size of T₂ it's very likely that when you test your classifer on T₂, it will appear much better than it is, just because most of the instances in T₁ aren't represented (by similar instances) in T₂. This is called overfitting to the training set. One way to mitigate it is to perform cross-validation, the most common type of which is k-fold cross-validation.
In k-fold cross-validation, you further partition T₁ to k partitions, or "folds", and then hold out each i'th partition, for i ∈ [1,k], use all the rest k-1 partitions as a training set and test on the i'th held-out partition _during training_. So you train your classifier on partitions 1 ... k minus i, test it on partition i, and repeat this process for all i, recording the performance (accuracy, F1, ROC etc, whatever your metric is). Then you choose the model that performed the best on your chosen metric.
And then you test it on T₂.
To avoid confusion between the k folds of T₁ that you use for testing your training models during cross-validation, on the one hand, and T₂, that you use for testing the model that performed best on cross-validation, on the other hand, we call the testing process performed on the k folds "validation" and each i'th subset of T₁ used for validation a "validation set". And we just call T₂ the "testing set".
The confusion arises because we do actually _test_ on sub-sets of T₁. But T₂ is always the "testing set" and it's never "seen" during training.
As to hyperparameter tuning, this is done _on the testing set_, i.e. T₂. This is A Very Bad Thing™ but there you go. Once you train a classifier and find out that it sucks on T₂, what do you do? Well, you tune the classifier's hyperparameters. Or do a grid search to automate the process. So eventually you overfit your classifier to the test set, because you now essentially have no "unseen" data instances in T₂ - the classifier didn't see the instances in T₂ during training but the trainer did, or, worse, the grid search did, and the classifier's hyperparameters were tuned according to that knowledge. How to avoid that, is a big question, but anyway that's what is done in practice, and the reason for that is that when you do Big Data, you end up needing so much data that despite having terrabytes of it, you never have enough.
That... that doesn't influence one of the presumed ways the NN categorizes images: the trend in bone geometry. The "blobs", while fuzzy, still largely retain the relative proportions to each other. Or, in other words, proportions of image elements are invariant for operations of scaling and of blurring.
The fact that trained neural networks cannot tell us why they give an answer and the best tool we have to explore that is to wiggle the inputs and see how the black box responds is a major concern for the whole space. Figuring out how to tag data with enough information to generate a "why" was an active area of research ten years ago and still is.
https://www.darpa.mil/program/explainable-artificial-intelli...
One of the points of building these systems is to do better than human-driven.
The practical one is that errors in a machine system scale, as do most things with machines. If I have a single bad X-ray tech who is applying the wrong medical process because I have a different race, for some reason, the damage that tech is doing is limited to whatever specific set of patients they are seeing. If a similar error occurs in a popular machine classification tool used widely by a hospital network, the damage is widespread. It is a plus that the machine can be corrected and the correction also scales, but with the (relatively speaking) stone tools we use to understand why a CNN makes its decisions these days, every fix risks breaking something else we're not testing for.
The first psychological reason is that machine learning systems break in "alien" ways. They don't make the kind of mistakes humans make... They make mistakes as a product of their machinery, which means it's much much harder to predict what those mistakes will look like for an average operator. As a frequent example, it's pretty rare for humans to misclassify human beings in photographs as apes, or to fail to recognize a face in an image because the skin is too dark. That's a failure mode that happens over and over again with image recognition systems.
And the second psychological reason is that humans don't trust machines to make human decisions yet. And that mistrust doesn't extend to other humans, even though we're incapable of cracking open another human's mind and understanding their thought process at the mechanistic level. It doesn't matter... we are the same organism and have a shared experience and empathy with them that we lack with machine recognition systems. It's semi-irrational, but it can't be wished away. A system for understanding why a machine makes decisions would be a step in the direction of addressing those concerns.
Perhaps hospitals that treat a disproportionate share of poor people (which themselves are disproportionately not white), tend to use a different brand of X-ray film, and that brand has different contrast ratios than that of the brand preferred by rich hospitals. Thus, they'd be detecting the different brand of X-ray film rather than anything about the patients themselves.
Of course, at this level it's still hard to imagine generating that 82% hit rate. But maybe there are multiple factors along these lines.
Most of us radiology folk abandoned film 20 years ago and went to digital systems (CR or DR). This doesn’t negate your query though, as vendors do have different technologies and their images do not look the same.