Some see this fact as problematic, and advocate that we should deliberate bias the results in order to create a more equal outcome. And the arguments of whether or not we should bias our models are frequently political in nature. The complaints that AI is biased is frequently because it produces results that some people find are problematic, but the underlying pattern it's identifying is true. It's really that the AI matches patterns without our cultural sensitivities about what topics we're supposed to tiptoe around.
The AI results are the way they are, because the data that is input tells the AI to spit out those results. In other words, the AI isn't telling you anything that a human didn't tell it to tell you. The valuable insights are insights that no human saw coming.
So let's take your industry as an example with which you can maybe better relate. Humans give the AI data that says that africans and men have higher recidivism. The AI then spits out results that says that africans and men have higher recidivism. Basically spitting back the information that we already gave it. OK, so far, so expected.
Now here's the thing, the valuable insight would be for the AI to suss out some counter intuitive result that no one would ever have gotten. The result that this particular african, or this particular man will have lower recidivism. That's the valuable insight. That insight saves the enterprise money that the enterprise would not have saved otherwise.
My point is that current AI's can't do that, which makes them not terribly useful. At the moment, they just parrot what any human would tell you anyway. So why pay 2 mil for that? Can you see what I'm saying? This problem is even more acute in diagnostic imaging, because the AI is not improving outcomes. So what, exactly, would we be paying 2 mil for? It's absolutely problematic for me to go out and put an enterprise on the hook to the tune of 2 mil for a black box that sits in a room, looks cool, and tells us things we already know. (In fact, the box would tell us things we told it to tell us.) We can save the money, by a 30 dollar digital recorder, and have it play the doctors' diagnoses back to the doctor after every patient meeting if that enterprise is really that much into the parroting.
My problem with the bias in the AI's I've seen is not political, it's practical. This stuff, as it is, is not terribly practical because it is too biased to offer any truly unique actionable insights that add value which was not there previously.
The allegations of bias in AI are frequently about inter-grouo differences in things like job advertisement, credit ratings, parole decisions, etc. which are not explicitly political but the allegations of often political in nature.
Sure, there's not much controversial about predicting tumor growth. But are the allegations of bias in AI about models predicting tumor growth? No, they're usually about the topics I listed above.
So when we're evaluating these systems, (again, systems that always fail due to bias), our only concern from a diagnostic viewpoint are patient-centric metrics. The most important being improving outcomes.
By way of illustration, let's suppose a woman goes for a mammogram. Now a radiologist and these AI's can all identify tumors in the resultant DX study at about the same point in said tumor's development. This leads to our current 2 and 5 year survivability rates. Consider thoughtfully here, the name of the game is not liberalism, or conservatism, the object in our game is to improve those survivability metrics.
Here's the thing though, due to the fact that the AI is biased towards things we already know, using it doesn't allow us to identify problems any earlier. Being able to spot issues earlier would improve 2 and 5 year survivability metrics, but current state of the art AI's won't do that. This is the problem in a nutshell.
The problem is not that the resultant data does not fit the "narratives" of our radiologists.
I'm not certain you understand that this is not a political argument. The AI's that vendors are attempting to apply towards problems in this domain simply don't improve healthcare outcomes. If they did, we would aid in deploying them. Without hesitation.
Now my assertion is that these AI's fail to improve healthcare outcomes due to the bias inherent in the data they use to train on. Again, that's not a political statement. It's just fact. The AI's seem mostly trained to internalize, and replicate as output, all of the well known patterns in the data. This is why these AI's are of severely limited use.