But it's a really important one and a lot of medical ML research doesn't seem to address in the proper sense. A very simple example: a ML model that classifies lung nodules as benign or malign, with 95% accuracy vs 70% accuracy of regular radiologists. Very good, right? But for actual, real world results, you need to see how the patient outcomes change. If the patients where the model and doctors disagree were going to have extra tests or followup regardless of what the model says, the model is not actually offering anything new despite the increase in accuracy.

So no, it's not absurd to say that models that work for a single type of test need to be very, very accurate to actually bring changes to the procedures that are worth the investment. What matters is whether they actually change patient outcomes, and sometimes it seems like the ML researchers barely consider it.