lukeor
There are lots of ways bias can still occur, like in who gets referred for scans, when they get referred, what the referrer writes on the request form, how the technologist takes the images (I could tell you some horribly racist stories about a few ultrasonographers I've worked with), and so on.
And all of this is based on previous work that AI produces bias (when trained on these datasets). If it was useful differences that drove AI learning about race, the models would not produce disparities. We went looking for how it is interacting with race because we already knew it was producing unacceptable outcomes. The big news here is that it is so easy to learn race that this effect is almost certainly not isolated to the systems tested so far.
1) race is not provided explicitly or intentionally during training, but medical practice is biased so it is reasonable to assume there is a signal in the data.
2) we know there must be a signal, since AI models learn it. The optimisation process should only discover features that correlate with the labels, but we see the ability to predict race in models trained to look for diseases like pneumonia (which do not appear different for people from different racial groups).
That paper shows that imagenet classifiers retain their ranking on data drawn from the same distribution as the training set. That isn't even close to the same thing as generalising to real world data. Medical AI shows us over and over again that truly out of distribution unseen data (external validation) is a completely different challenge to simply drawing multiple test sets from your home clinic.
Again, I don't actually think imagenet was as problematic as other competitions, but there is better evidence for that (not the least of which is that for the first half of imagenet's life, the differences in models were large, the number of tests was cumulatively fairly small, and the test set was huge: ie what I wrote supports imagenet as fairly reliable).
I've fallen victim of getting a Twitter bump, and assuming that people know I'm not anti-ML.
The blog post is meant to be educational, not argumentative. Since it has got wider exposure I'll do a follow up to clarify my position on imagenet.
No statistician would do what I did for a formal publication, but I think what I did gets the point across.
What the picture was trying to say is that, within a given year, the "winner" becomes less likely to be truly better than the second place team. Alexnet was clearly better than the alternative, even with Bonferroni adjusted significance thresholds. Less so by 2016/17.
I'm writing a follow up on imagenet in particular to address some of the nuance. It is very clearly not a representative example of ML competitions, but the same effects still apply to some extent (imo).
I'm just getting down to writing some posts about the big question the New Yorker article introduces, but doesn't really make any real progress with: will machines actually replace doctors.
Hope you enjoy them :)
I think I would say that the use of private datasets isn't actually a problem though, the problem is that the methods and analysis in many of the papers is horribly flawed. I could reasonably trust a paper with good methodology - all of medical research is on private data essentially. But the majority is, as you say, pure hype.
I actually wrote another blog post a bit back that was popular on some simple rules to separate hype from reality in medical AI work - https://lukeoakdenrayner.wordpress.com/2016/11/27/do-compute...