There's a difference between: not ideal to tackle and not ideal to build a business around.
It is absolutely ideal to tackle which will hopefully build confidence in a system that can then be expanded to solve harder problems. If you can't even solve this problem I don't see how you're getting public confidence in the harder ones.
The third world might need more effective training, so they can convert more cheap labor into specialists, but I don't see why per se they would prefer to use a computer sold at Western prices to a local specialist.
That's empirically not true; the first such product approved for clinical use entered the market in the late 90s, and they've had small-to-medium success since. Definitely none of them changed things radically, but it's pretty common CAD functionality now and has both detractors and champions.
> Radiologists can fly through those exams with high quality,
This is also not entirely true, one of the biggest problems for screening (not diagnostic) programs is that the false negative rate for radiologists is high. They do fly through these scans, but that's to make it economically feasible. Also, the average radiologist performance on screening mammo is not very good - most have to do it regularly (probably 100s per month) to do it well.
You are right that augmentation is the more plausible path to clinical success, but that's been what most of the clinical systems aim to do anyways, provide a sanity check on the radiologist.
E.g. If a system could tell me which 967 results out of 1000 are normal, and send the other 33 for human review.
True positives require solving every problem in the solution space. True negatives only require solving the most frequent problems.
Plus--wouldn't the software company be liable? (I don't think the tech is ready for office use, but when it is--I still think it will be a battle.)
Kinda like all medical devices. Whenever there's a problem, the equipment is looked at first?
Personally, I don't think the AMA wants computers taking away their doctor's precious income, and will only acquiesce when the tech makes the doctor more money, or is so good politicians/insurance companies start demanding it.
But IMHO medical law in general has a big device / software blind spot. It's become decent at figuring out liability for humans, while permitting normal operations.
But all that machinery and case law doesn't really exist for black box software systems (no human in the loop). And if there's one thing that medical providers and insurance companies hate, it's unbounded liability.
Often, the ML provides an extra layer that slows things down, even if it's working at state of the art AUROC or whatever we're trying for.
There's also the fact that radiologists get paid a pretty decent salary. If I could devise a system that can do a highly paid worker's job faster, cheaper, and at a similar level of effectiveness, why wouldn't I? It also opens up that skilled worker's schedule and allows them to tackle more difficult to interpret scans.
The point I'm making is that focusing on mammograms is not the way to do that. It's already a very optimized specialty. If AI researchers worked more closely with practicing radiologists (not academic ones) I imagine there's all sorts of areas where existing technology today can improve things.
That's almost certainly true, and surely there must be people in the medical industry already working on this. In hospitals, there are internal organisations that vaguely attempt to link up technology workers with practicing doctors to improve medicine using technology. One of my most competent former developer colleagues retrained as a doctor and worked in medical digitisation, as well as being a practicing doctor.
However:
* Often coming at a problem from another angle and ignoring existing professionals can be useful too. Existing workers are very often stuck in their ways and unwilling to accept the way they look at a problem may not be the best. I have heard from many in medicine, that doctors are extremely distrustful and hesitant towards technology, as well as fiercely protective of their jobs.
* The medical industry is internally highly dysfunctional and trying to work with practicing doctors can be extremely politically difficult. The hospital bureaucracy often actively tries to prevent practicing doctors from working directly with technologists, preferring instead to silo them and direct all communication via senior management - who are often neither technology nor medical experts. Consequently it seems more likely that major changes will come from outside.
* AI has occasionally (rarely?) shown great success in solving problems without needing domain experts. See AlphaGo.
Games are always orders of magnitudes easier then analyzing reality. They are highly constrained to the narrow domain and space of possibilities. Reality doesn't. There is work on understanding physical laws by the AI but it's very long path and they are much less successful than their artificial reality colleagues.
In game domains we have the ability to generate infinite meaningful training data and have easily measured outcomes. Waymo is generating near infinite data for driving, and doing quite well with it. In medicine, we have piddly ass datasets because of privacy concerns. 'Reality is hard' had nothing to do with it.
This isn't going to be a commonly done thing.
This is one of the areas where working with radiologists actually makes things easier. Radiology groups are normally independent practices that contract with multiple hospitals- they link in with hospital systems but mostly run their own thing. They also tend to have really solid IT departments and enough funding to make things work. Working with hospitals is a nightmare though, as you say.
It's the reason professional baseball is experimenting with electronic strike detection, despite the fact that umpires are very accurate (~94%) calling pitches.
I get that there are useful applications of AI that could augment a radiologist. I'm simply pointing out that if one could completely remove a radiologist from the equation, that could be a super helpful tool. I suppose my view is that removing/limiting the human element in routine inspections is a very worthwhile goal.
There's also the side benefit of increasing trust for these technologies. I think a lot of people are rightfully scared of AI technologies; showing that they can be as effective as a radiologist for routine screenings would be very beneficial.
You are suggesting something that's against the groupthink for that profession. You can't graduate from an MD program while holding these thoughts.
I'm being 100% serious here.
By your logic I must not exist then.
Like who do you think is labeling your training images?
Look at the authors of pretty much all cited studies in that meta-analysis. Sure see a lot of MD and MD-PhD there.
to have them read an AI's diagnosis before having to assess a mammogram themselves to confirm the AI's finding is adding an unnecessary step to their workflow.
I think you're disagreeing with a point that the parent comment didn't make.
It's worthwhile in general to continue AI research. The properties of this application make it ideal for testing and progressing AI technology. But if AI doesn't end up being used in this particular area in the short or medium term, that's another matter entirely.
But those are also harder for software based systems.
But yes, augmenting human intelligence instead of substituting it often does work better.
That to me sounds like an ideal automation use case. You have an entire class of highly skilled professionals performing a repetitive task. Sure, they may be able to do it quickly, but I imagine they must have to do a bunch of them each day.
You could imagine a world where hospitals didn't have radiologists on staff, but rather they acted in a consulting capacity for difficult cases.
Walking on variable surfaces is also a highly repetitive task and automation is yet to master it.
For some reason CS people fixate on radiology for automation just because it is imaging, but there's a lot of context behind it. There's a reason a radiologist/pathologist is called a doctor's doctor, and no one in the field is worried about automation.
For perspective: Training to be a radiologist is 5+1 years, your family doc trains for 3.
Precisely because it's imaging. Training data is abundant, and has the potential to be well labelled. And one of the most active field of AI research is... computer vision. So it's no wonder the low hanging fruit would be medical imaging.
> There's a reason a radiologist/pathologist is called a doctor's doctor, and no one in the field is worried about automation.
Spoken like Garry Kasparov.
> For perspective: Training to be a radiologist is 5+1 years, your family doc trains for 3.
What I love is how the immediate knee jerk reaction isn't to explain why the ML approaches won't work but to immediately retreat behind gatekeeping, in this case the tittle and number of years of schooling.
Replacing them with AI would seems like a straightforward process.
What behind the scenes context would make it impossible?
The MD context. One first needs to train AI to perform the job of a general MD before you can get into stuff like radiology (that is, their real job, not what some novice CS grads, or not so novice AI experts like Hinton imagine it to be - I.e. not segmenting things into funky shapes or running some funky black box magic that spits out "tumor/not tumor" with no context whatsoever, no. Actually diagnosing real people, where a life is on the line, and if you fuck up enough times, your career).
That's true. Waymo had to train their AI to perform the job of a firefighter for them to recognize a fire hydrant and a firetruck.
> I.e. not segmenting things into funky shapes or running some funky black box magic that spits out "tumor/not tumor" with no context whatsoever, no.
... But that's the goal right? There's a tumor right there at these coordinates or not.
These questions and their relevance changes for every individual case, and while each question by itself may be approachable with AI, getting a detailed and relevant report without meaningless noise from an AI ensemble is a very very hard problem.
These are all interesting problems where I could see an AI struggling. I guess the next step, once tumor identification becomes a solved problem, will be to train the AI on treatment data and follow-up, ie, this is an example where there was spread to lymph nodes.
Interesting times ahead!
Part of it.
Human doctors will not only tell you if there's something funky in the image, but will also interpret it in light of a patient's medical history, symptoms, possible diagnoses, etc.
Subtle shading near some structure involved in one of two possible conditions might be very important, but an obvious cyst in an unrelated organ likely means nothing. People are weird close-up!
And yet....
The medical field is notorious for delivering "efforts", not results (unlike engineering). No SLA here.
Right now much of the mammogram reading is extremely straight forward. Radiologists can fly through those exams with high quality, especially when you consider the macros they have setup that allow them to write most of the report with with a two second voice command.
This is one of the biggest things radiologists complain about- from their perspective most AI is solving problems that aren't actually useful to radiologists. I think the companies that will do best are the ones that augment radiologists with a focus on making them more accurate and efficient, rather than trying to cut them out of the loop (I'm a bit bias though as my previous company, Rad AI, is taking the augmentation approach and it has worked out rather well).