Right now much of the mammogram reading is extremely straight forward. Radiologists can fly through those exams with high quality, especially when you consider the macros they have setup that allow them to write most of the report with with a two second voice command.
This is one of the biggest things radiologists complain about- from their perspective most AI is solving problems that aren't actually useful to radiologists. I think the companies that will do best are the ones that augment radiologists with a focus on making them more accurate and efficient, rather than trying to cut them out of the loop (I'm a bit bias though as my previous company, Rad AI, is taking the augmentation approach and it has worked out rather well).
There's a difference between: not ideal to tackle and not ideal to build a business around.
It is absolutely ideal to tackle which will hopefully build confidence in a system that can then be expanded to solve harder problems. If you can't even solve this problem I don't see how you're getting public confidence in the harder ones.
The third world might need more effective training, so they can convert more cheap labor into specialists, but I don't see why per se they would prefer to use a computer sold at Western prices to a local specialist.
That's empirically not true; the first such product approved for clinical use entered the market in the late 90s, and they've had small-to-medium success since. Definitely none of them changed things radically, but it's pretty common CAD functionality now and has both detractors and champions.
> Radiologists can fly through those exams with high quality,
This is also not entirely true, one of the biggest problems for screening (not diagnostic) programs is that the false negative rate for radiologists is high. They do fly through these scans, but that's to make it economically feasible. Also, the average radiologist performance on screening mammo is not very good - most have to do it regularly (probably 100s per month) to do it well.
You are right that augmentation is the more plausible path to clinical success, but that's been what most of the clinical systems aim to do anyways, provide a sanity check on the radiologist.
E.g. If a system could tell me which 967 results out of 1000 are normal, and send the other 33 for human review.
True positives require solving every problem in the solution space. True negatives only require solving the most frequent problems.
Plus--wouldn't the software company be liable? (I don't think the tech is ready for office use, but when it is--I still think it will be a battle.)
Kinda like all medical devices. Whenever there's a problem, the equipment is looked at first?
Personally, I don't think the AMA wants computers taking away their doctor's precious income, and will only acquiesce when the tech makes the doctor more money, or is so good politicians/insurance companies start demanding it.
But IMHO medical law in general has a big device / software blind spot. It's become decent at figuring out liability for humans, while permitting normal operations.
But all that machinery and case law doesn't really exist for black box software systems (no human in the loop). And if there's one thing that medical providers and insurance companies hate, it's unbounded liability.
Often, the ML provides an extra layer that slows things down, even if it's working at state of the art AUROC or whatever we're trying for.
There's also the fact that radiologists get paid a pretty decent salary. If I could devise a system that can do a highly paid worker's job faster, cheaper, and at a similar level of effectiveness, why wouldn't I? It also opens up that skilled worker's schedule and allows them to tackle more difficult to interpret scans.
The point I'm making is that focusing on mammograms is not the way to do that. It's already a very optimized specialty. If AI researchers worked more closely with practicing radiologists (not academic ones) I imagine there's all sorts of areas where existing technology today can improve things.
That's almost certainly true, and surely there must be people in the medical industry already working on this. In hospitals, there are internal organisations that vaguely attempt to link up technology workers with practicing doctors to improve medicine using technology. One of my most competent former developer colleagues retrained as a doctor and worked in medical digitisation, as well as being a practicing doctor.
However:
* Often coming at a problem from another angle and ignoring existing professionals can be useful too. Existing workers are very often stuck in their ways and unwilling to accept the way they look at a problem may not be the best. I have heard from many in medicine, that doctors are extremely distrustful and hesitant towards technology, as well as fiercely protective of their jobs.
* The medical industry is internally highly dysfunctional and trying to work with practicing doctors can be extremely politically difficult. The hospital bureaucracy often actively tries to prevent practicing doctors from working directly with technologists, preferring instead to silo them and direct all communication via senior management - who are often neither technology nor medical experts. Consequently it seems more likely that major changes will come from outside.
* AI has occasionally (rarely?) shown great success in solving problems without needing domain experts. See AlphaGo.
Games are always orders of magnitudes easier then analyzing reality. They are highly constrained to the narrow domain and space of possibilities. Reality doesn't. There is work on understanding physical laws by the AI but it's very long path and they are much less successful than their artificial reality colleagues.
In game domains we have the ability to generate infinite meaningful training data and have easily measured outcomes. Waymo is generating near infinite data for driving, and doing quite well with it. In medicine, we have piddly ass datasets because of privacy concerns. 'Reality is hard' had nothing to do with it.
This isn't going to be a commonly done thing.
This is one of the areas where working with radiologists actually makes things easier. Radiology groups are normally independent practices that contract with multiple hospitals- they link in with hospital systems but mostly run their own thing. They also tend to have really solid IT departments and enough funding to make things work. Working with hospitals is a nightmare though, as you say.
It's the reason professional baseball is experimenting with electronic strike detection, despite the fact that umpires are very accurate (~94%) calling pitches.
I get that there are useful applications of AI that could augment a radiologist. I'm simply pointing out that if one could completely remove a radiologist from the equation, that could be a super helpful tool. I suppose my view is that removing/limiting the human element in routine inspections is a very worthwhile goal.
There's also the side benefit of increasing trust for these technologies. I think a lot of people are rightfully scared of AI technologies; showing that they can be as effective as a radiologist for routine screenings would be very beneficial.
You are suggesting something that's against the groupthink for that profession. You can't graduate from an MD program while holding these thoughts.
I'm being 100% serious here.
By your logic I must not exist then.
Like who do you think is labeling your training images?
Look at the authors of pretty much all cited studies in that meta-analysis. Sure see a lot of MD and MD-PhD there.
to have them read an AI's diagnosis before having to assess a mammogram themselves to confirm the AI's finding is adding an unnecessary step to their workflow.
I think you're disagreeing with a point that the parent comment didn't make.
It's worthwhile in general to continue AI research. The properties of this application make it ideal for testing and progressing AI technology. But if AI doesn't end up being used in this particular area in the short or medium term, that's another matter entirely.
But those are also harder for software based systems.
But yes, augmenting human intelligence instead of substituting it often does work better.
That to me sounds like an ideal automation use case. You have an entire class of highly skilled professionals performing a repetitive task. Sure, they may be able to do it quickly, but I imagine they must have to do a bunch of them each day.
You could imagine a world where hospitals didn't have radiologists on staff, but rather they acted in a consulting capacity for difficult cases.
Walking on variable surfaces is also a highly repetitive task and automation is yet to master it.
For some reason CS people fixate on radiology for automation just because it is imaging, but there's a lot of context behind it. There's a reason a radiologist/pathologist is called a doctor's doctor, and no one in the field is worried about automation.
For perspective: Training to be a radiologist is 5+1 years, your family doc trains for 3.
Precisely because it's imaging. Training data is abundant, and has the potential to be well labelled. And one of the most active field of AI research is... computer vision. So it's no wonder the low hanging fruit would be medical imaging.
> There's a reason a radiologist/pathologist is called a doctor's doctor, and no one in the field is worried about automation.
Spoken like Garry Kasparov.
> For perspective: Training to be a radiologist is 5+1 years, your family doc trains for 3.
What I love is how the immediate knee jerk reaction isn't to explain why the ML approaches won't work but to immediately retreat behind gatekeeping, in this case the tittle and number of years of schooling.
Replacing them with AI would seems like a straightforward process.
What behind the scenes context would make it impossible?
The MD context. One first needs to train AI to perform the job of a general MD before you can get into stuff like radiology (that is, their real job, not what some novice CS grads, or not so novice AI experts like Hinton imagine it to be - I.e. not segmenting things into funky shapes or running some funky black box magic that spits out "tumor/not tumor" with no context whatsoever, no. Actually diagnosing real people, where a life is on the line, and if you fuck up enough times, your career).
That's true. Waymo had to train their AI to perform the job of a firefighter for them to recognize a fire hydrant and a firetruck.
> I.e. not segmenting things into funky shapes or running some funky black box magic that spits out "tumor/not tumor" with no context whatsoever, no.
... But that's the goal right? There's a tumor right there at these coordinates or not.
These questions and their relevance changes for every individual case, and while each question by itself may be approachable with AI, getting a detailed and relevant report without meaningless noise from an AI ensemble is a very very hard problem.
These are all interesting problems where I could see an AI struggling. I guess the next step, once tumor identification becomes a solved problem, will be to train the AI on treatment data and follow-up, ie, this is an example where there was spread to lymph nodes.
Interesting times ahead!
Part of it.
Human doctors will not only tell you if there's something funky in the image, but will also interpret it in light of a patient's medical history, symptoms, possible diagnoses, etc.
Subtle shading near some structure involved in one of two possible conditions might be very important, but an obvious cyst in an unrelated organ likely means nothing. People are weird close-up!
And yet....
The medical field is notorious for delivering "efforts", not results (unlike engineering). No SLA here.
The same issues bedevil prostate cancer screening.
But since screening has been framed as caring for women, pointing out the flaws with screening is automatically seen as hating women.
While Reagan did a lot to try to defund cancer research regardless, the first lady's mastectomy drew a lot of attention to a previously taboo topic.
[0] https://www.nytimes.com/1985/07/16/us/reagan-s-illness-medic... [1] https://www.latimes.com/archives/la-xpm-1987-10-18-mn-15261-...
Back when I was a life scientist the best advice I got was to work on diseases the family members of congresspeople had.
This is not a "weird relationship" thing, rather women got organized and: https://en.wikipedia.org/wiki/Breast_cancer_awareness
It’s more broad than that, it’s the relationship with women’s health as a whole.
If you're really interested, watch some older Dutch movies to see the normalness of naked, like Turkish Delight [0]. Or even have a closer look at the relation between the Professor and Raquel in the recent Casa the Papel. It's different from US series. More respectful, mature wrt women if you ask me. More emphasis on intelligence. Or on a beautiful woman in her 40s with normal wrinkles. The US, like on many other topics, seems to be polarized, caught between the hyper sexuality of Cardi B and the prude nature of American culture in general.
[0]: https://en.wikipedia.org/wiki/Turkish_Delight_(1973_film)
Fixed that for you.
So our best shot from a public health perspective is to say "Here's what we recommend for everybody" and pay for that.
All screening programmes have two difficulties, which must be balanced against the benefit, and this trade is somewhat personal, so when the balance is quite fine the arguments can be vociferous as a result.
1. The screening itself may seem unpleasant. One woman may find it a very mild annoyance, a drive ten minutes out of her way, the staff are very pleasant, the scan itself is far less traumatic than a bra fitting, and she receives easy to understand results after not very long and isn't anxious about them; but for another maybe it's an hour's bus journey to the city hospital, the staff there are short-tempered and say she has the wrong paperwork, then another hour in a queue, she feels like she's just meat, squashed around for the convenience of the machine for what seems like forever, and then after anxiously waiting for what seems to be too long the results are confusing to her and she has to have a friend interpret them.
2. Over-treatment is always a problem. Screening by definition detects something that isn't causing noticeable symptoms. If you have a noticeable lump, or mysterious bleeding, you don't need screening you need a doctor's appointment. So a positive screening result might be nothing important. However either you've now got the burden of a diagnosis you ignored or, you accept the medical advice and are treated, even though it's possible (not likely, but possible) that you would have been just fine without treatment.
So, screening programmes are set up based on guessing how to trade these factors plus a third, how much should we spend on this medical intervention? After all, in some sense every dollar doing breast cancer screening is a dollar you don't have to cure blindness in poor orphans (or of course, to bomb somewhere)
If your experience of a screening programme is that it's a minor inconvenience at most, and yet you know people who died of undetected disease, more screening seems like a no brainer. Particularly if you live somewhere where screening stops at age 50, and somebody you know died of undetected disease aged 54, you might reason that the screening should go to age 55 or 60 to detect such cases, no matter the public cost.
On the other hand if your experience is that it's an awful ordeal even when negative, and you know people who spent their last years horribly scarred by surgery as a result of suspected disease but then they died in their sleep from something else anyway, you may feel that there's already too much screening and it should be trimmed back, not to save money but the extra money for other programmes is welcome.
I certainly don't worry about the cancer I could develop as a result of an xray in a regular dental cleaning (even though that's a possibility).
However if you are a doctor trying to figure out how often to screen women, the danger if xrays is a strong reason to not do it too often. Daily screenings would catch breast cancer a lot sooner on average, but just isn't worth the risk even if the screening was otherwise free.
You may want to consider finding a different medical professional!
Our hospital bought an AI stroke detector (viz.ai). The "AI" part is laughable but it allows the hospital to collect some extra HHS/medicare fee for using "imaging algorithms" or something. I suspect that company is potentially using anonymized scans to continue training their data, because the initial product was trained on some tiny sample of brain CT scans, like under 500.
The one plus for actual physicians and nurses is that at least they wrote a non-sucky PACS imaging interface for the iPhone so we can just pull up the plain scans and view them easier.
Why would that even pass the FDA smell test?
https://en.wikipedia.org/wiki/Mammography_Quality_Standards_...
This looks good from 70,000 ft, but in practice you need labels much more specific than BIRADS.