Alternatively, imagine a proactive medical world, in which preventative screenings are commonplace. Currently, the implementation of routine screenings without any complaints is prohibitively expensive on a large scale. This is because it requires manpower, and manpower is prohibitively expensive and the expense of manhours needs to be justified by a medical practitioner. However, AI can help in this proactive medical world by reducing the number of hours real people are looking through data to detect problems of patients, reducing the cost of routine screenings at large. Again, this wouldn't replace doctors, as you'd still need a specialist to analyze any positive hits, but it differs from your scenario in which the AI diagnosis seems redundant.
So, when preventative medical practices are more prevalent, the mass routine screening procedures will need help from machines to keep it cost effective, and that I believe is where this technology will find its application.
What you do want out of AI is to flag areas of interest in imaging for example and help identify when records are at risk of being incorrectly normalized. Ideally, even if the end effect is marginal (say bumping accuracy from 80% to 90%), if it enables a workflow that decreases the exhaustion and frustration of the doctor you will want that in place.
Of course it could just as well be used as an excuse by management to increase any given doctor's throughput, so it might not work as you would want.
Over time as AI improves, more and more diagnoses can look like this.
https://www.cochrane.dk/news/new-study-finds-breast-cancer-s...
Seems like it should help, but in practice leads to over-diagnosis.
For example - Cancer rates jumped in Korea after screening with no impact on patient outcomes [1]. There are several others.
[1] Lee, J. H., & Shin, S. W. (2014). Overdiagnosis and screening for thyroid cancer in Korea. The Lancet, 384(9957), 1848.
This is how you end with false positives and over-diagnosis.
For lung cancer screening, NLST showed a 20% reduction in mortality and now NELSON has shown even stronger results in Europe.
This “all screening is bad” is FUD in the medical field, frankly. Yes it has to be studied and implemented carefully, but to make blanket statements about screening as a whole is factually incorrect.
Broad-based population screenings as the parent comment suggests, in my opinion, are.
I'm yet to see any clinically-valid distinguishing aspects that would suggest AI would add value to screening. Curious to hear evidence that drives your optimism of human+AI.
Just to state, the NELSON study [1] focuses on high-risk segments. Their paper also recommends a "personalized risk-based approach" to screening. This seems reasonable.
You certainly said that screening leads to over diagnosis.
I think for screening, the best results are probably the upcoming prospective study from Kheiron.
https://www.kheironmed.com/news/press-release-new-results-sh...
will show stronger performance. But Kheiron appears to be ahead as far as proving the value of the tool since they have actually validated prospectively.
Truly effective preventative measures are the apex achievement of medical science, and have simply deleted an unimaginable amount of human suffering from our modern lives. Vaccines and sanitation are the best examples. They are astoundingly cost effective measures, and are so good that in many ways they are unimprovable in any significant way. 2020 is yet another example of how important vaccine technology is to every single person on this planet.
Screening is nowhere near as beneficial as actual prevention. It is expensive, labour intensive, requires behavioural modification, definitely harms a significant proportion of people due to false positives, and in controlled trials only modestly improves hard clinical outcomes under the most charitable assumptions of compliance and follow up care.
You are proposing a model of 'high-touch' medicine, where people have a raft of continuously administered screening tests for a long list of conditions. This could only ever be applied to a small proportion of the world's population, and would require highly motivated and well educated patients. In my opinion, spending a day in a primary care medical clinic would disabuse you of the notion that this is a feasible or desirable outcome.
I want to point out, that any serious machine learning researcher is not oblivious to this, despite the deep learning boom suggesting to the contrary. Modern methods have shown that we are capable of building predictors with surprisingly complex representations, that can solve large-scale downstream tasks. i.e. our models are "flexible" enough.
The next challenge is whether they favor the "right kind" of solutions. For instance, Convolutional Neural Networks (CNNs) are architecturally just sparse version of Fully-Connected Neural Networks. Why is it then that CNNs perform far better on images? A key reason is that "inductive biases" afforded by MLP aren't strongly favored towards images. Another instance of this is the covariance functions used is Gaussian Processes - the Squared Exponential Kernel is very flexible and can in principle fit anything possible. Nevertheless, if the problem has specific structures, say periodicity, one better use the Periodic Kernel because it's inductive biases rightly align with the kind of solutions we expect.
> The lesson is : physics beats AI.
As a consequence, the single biggest reason physics would beat a generic AI in the short-term is precisely due to our ability to explicitly provide inductive biases that align with our expectations from the physical system.
We haven't found the secret sauce for every possible system in the universe. I don't think we can, either. But what we can do is devise ways to "control" such inductive biases we can encode in machine learning systems, which align with our expectations of the way the system should behave.
For example, viz.ai has a solution to help get brain bleeds spotted to the eyes of surgeons more quickly. It is deployed and has cut the average length of stay in the neuro ICU significantly
https://mobile.twitter.com/viz_ai/status/1314710308603133953
I work at Caption Health, where we are enabling novices to take echocardiography scans. The doctors who work with our technology found it extremely useful to help diagnose cardiac involvement during covid.
https://captionhealth.com/education/
As much as I have respect for the expertise of medical doctors, I would ask that you have respect for folks working to apply AI in medicine.
I mainly intended to underline the fact that we (doctors) were promised a revolution in healthcare (AKA : to disappear) and we ended with diagnostic scores.
However, I gladly admit that I exaggerated and that AI technologies can be helpful in some cases, of course.
I think it’s the classic thing where it’s overestimated in the short term and underestimated in the long (longggg) term.
My sense is that for AI to have the full impact it will one day reach, it will take rethinking medical care entirely with online machine learning and data at the core of how decisions are made.
ML was able to revolutionize how ads are delivered (for better or worse, but at least reaching the objectives of those who deployed it) because you can update and deploy the models multiple times a day.
If we can one day get to a world like that where an ML mode is constantly learning and updating itself, and has seen far more patients than any individual doctor, then maybe we will see the sorts of bigger shifts that were imagined shortly after we started to see ML surpass human ability on long standing difficult tasks like object recognition.
Getting there is a long, long road where we need to learn to work together with AI and figure out where the holes are in terms of robustness, earn trust over years of successful deployment, and figure out how to properly put safety bounds around more frequent updates to models.
"I'm an eye surgeon and self-taught machine learning practitioner, I started to learn Python in 2016 when the deep learning hype was at his highest."
with:
I'm a [machine learning researcher] and self-taught [ophthalmologist], I started to learn [ophthalmology] in 2016 when the [clinical medicine] hype was at his highest.
In this hypothetical situation, I bet you would instantly discount what I would have to say about ophthalmology because I clearly would not have the depth or experience to have an informed opinion on ophthalmology.
Over the past few years with the ML hype, I have noticed quite a few clinicians who have self taught some deep learning methods claim expertise in the subject area (not targeting you, a general observation). I feel like many clinicians do not understand the breadth of machine learning approaches. There is just so much to know! from robust statistics, non-parametric methods, to kernel methods. Deep learning and deep generative models are by no means the only tools at our disposal.
I absolutely agree with you though. Applied machine learning practitioners have been over selling their accomplishments -- which I believe is detrimental to progress in the field.
I would highly encourage you to collaborate with ML researchers who have spent a decade or more working on hard problems. From the other side, I can tell you I gained a lot discussing ideas with domain experts (neurologists, radiologists, functional neurosurgeons). They have insights that I could never have picked up by self teaching.
One the other hand, ML researchers with a deep knowledge expertise are extremely hard to find, even among statisticians / programmers. I suppose that the people with a real expertise are working on their own startup or in FAANG.
This leads to a situation where the medical research involving ML is largely without interest or full of bias. It is easy to spot in the literature.
On a positive note, I'm so glad that clinicians are taking interest in ML! As a practicing ophthalmologist, the fact that you were able to self teach is really impressive! I do know that a lot companies are looking for people like you, who have clinical experience. If you are interested you should explore roles/potential collaborations with some of these health research teams in tech.
The former would be quite difficult without much background in ML/Computer Vision (you would have to spend some time self-teaching basics of ML/Deep Learning and the pre-reqs for those — Basic Linear Algebra and Probability).
The latter is doable. I would recommend a very hands on approach. Pick some computer vision object classification tutorials and code them up (using a high level library). Make a mind map of the concepts and look them up as and when you’re unclear about a concept. Then move on to replicating some well cited, peer reviewed papers. Often papers will have their code on GitHub. Try and relocate their results on their dataset. After this you would have the basic working knowledge to modify the algorithm slightly for your specific use case.
Looks like it's a new model, I have no idea if they already have any ML models yet. There's also some database work.
I'm finishing a Masters degree in Computational Physics, so Linear Algebra and Probability shouldn't be an issue. (We also have an Image Processing and Analysis course.) I guess that's why they contacted us despite the fact that we don't have any ML training ?
Yeah, this is basically what I thought to do, but thank you for your advice !
Also, might be useful to took at webpages of some researchers in this space and courses they teach [1,2].
[0] https://web.stanford.edu/~hastie/ElemStatLearn/
[1] https://scholars.duke.edu/person/dunson
[2] https://www.cs.princeton.edu/~bee/Funny (but I guess expected) to see the Markov Chain Monte Carlo method that we very recently learned in that book's table of contents ! (Unless it's another MCMC ?)
What do you think of the relative success of Diabetic Retinopathy (DR) diagnostic models, especially the FDA approval of the clinical trials Digital Diagnostics (formerly IDxDR) [1]? Their approach to the model architecture was slightly different from the black-box approach of other labs, wherein IDxDR's model is trained to look for clinically relevant indicators for DR. Is that a more likely route for future diagnostic AI models?
[1]: https://dxs.ai/newsroom/pivotal-trial-results-behind-the-fda...
So to answer you directly yes, it's a better route until we have better information available. But it's also the wrong route to take in the long term. It would be far better to attempt to produce better supporting information.
However, I would like to see the performances of the algorithm on different fundus camera. It is also important to realize that diabetic retinopathy classification is very easy to learn, to the point that if the screening is such a problem, it is easier to ask the person that takes the pictures (In France, a nurse or an orthoptist) to phone the doctor when he/she sees something strange on the eye fundus.
For companies with the enterprise expertise it is difficult to enter with the right mindset because sales cycle ib healthcare is way too long.
Confirm that that major botlenecks are indeed getting data in shape for modelling. Feature engineering is key and domain specific. Forget brute force approaches like DL.
Also most AI practicioners in the field seem to ignore that doctors dont need to know who is sick but who is actionable. Its a completely different game.
The technology is fascinating, and there has been a lot of interesting and clever innovations. But the value add for healthcare is pretty small in the end. The value add for the actual patient is even more miniscule. There is some misunderstanding about what doctors actually do and what patients want doctors to do. Fundamentally, a patient wants to see a doctor that is going to take responsibility for their medical problem, and ideally their health. When this is achieved, the outcomes are usually optimal under constraints, and the clinician and patient are both satisfied. 'AI' as it is known today could never do this. Even if someone unveils superhuman general AI tomorrow, it wouldn't be able to do this.
There are various niche applications of machine learning which employ interesting technology in image interpretation, and this will be a useful addition to the suite of tools already present, particularly where non-specialists need to look at a picture or radiological image, and for screening.
But otherwise I expect that in 20 years time we will still be saying that EMR data is too difficult to deal with, and uptake of machine learning based software will be very low. We still don't have useful automated reporting of electrocardiograms, for example. So even sophisticated curve fitting seems to have trouble fitting these literal curves in a way that impacts clinical care.
I had a similar revelation. I sat through an AI for health presentation and basically asked “ok, so you take a data set then try and find a set of rules that accurately describes...like a linear regression?”
As you said, it’s more sophisticated than that, but in essence, yes, it’s fitting a curve to data.
Current systems are limited to ingesting input and providing output.
A big part of medical diagnosis, however, is to do follow-up exploration based on results of a previous examination.
This is a big part of "intelligence" that's still missing entirely from all approaches that I'm aware of, i.e. the ability to ask further questions/request data based on preliminary results from previous inputs.
But intelligence isn't just a "brain in a jar" situation. Intelligence requires interaction with the environment - you'd never be able to tell whether a Boltzmann Brain is intelligent from just observing it, for example.
1. Technician writes down their diagnosis
2. They submit it to the system
3. AI comes with its own analysis
4. Technician sees the outcome, they can update their assessment
5. Everything is saved into the system
If one of technicians has too much errors in their initial assessments, it should raise a concern.
Will result in exactly what I described above.
> If one of technicians has too much errors in their initial assessments, it should raise a concern.
People will refuse AI oversight if there are associated sanctions. People will make every effort to game the system. Following that, you'll be left with:
a. Pay techs more, so they accept the new working conditions.
b. Fire all techs and make do with a (potentially suboptimal) AI system.
Yes, this is very much gate keeping at work.
Paper charts everywhere. Post it in some files.
Wonder how much ML could help with mammogram. If you ever read one of these, you mostly use symmetry and check the previous ones to spot what changed.
Working conditions in medicine are universally poor, frankly. The flexibility during education and training is close to zero. You probably will need to move somewhere else as part of training or work. This can uproot your life and family depending on obligations. Some fields have very prolonged training periods and are very competitive (eg surgical subspecialties). There is a lot of bullshit in medicine, due to heirarchies in large institutions, rigid and nonsensical regulations, issues with billing etc... you have to have the personality to stomach this. Conversely, there is also the fact that when you are in the room with the patient and the door is closed, it is just you trying to help another person, despite everything. This is sacred, and can be intensely rewarding even in dire circumstances, an exercise in mutual gratitude you just won't find elsewhere.
The other thing I would say is that some fields in medicine are probably not something you should spend your whole life doing, at least not full time. It is just too emotionally exhausting, no matter what you do to deal with it. Not saying it is impossible to do it for 40 years, but it is a risk, you definitely sustain some kind of damage. I suspect this is responsible for 2/3rds of the seemingly outrageous negligence/misconduct cases you hear about.
I am strongly attracted but I feel maybe it's mostly illusory.
I'm quite surprised by this. Doesn't each AI tutorial start by stating that very thing?
AI is not fundamentally different from a simple linear regression
This is my biggest complaint with the EMR systems I've used and I've always wanted to improve this. I wonder if fellow doctors would be okay with using a simple structured language to describe data in an EMR.
For example:
Height: 175 cm
Weight: 70 kg
Ethnicity: white
Age: 40 years
Creatinine: 0.9 mg/dl
Doctors already do this work, we even input this exact same data into calculator apps. The innovation would be recognizing this data in the EMR text and doing it automatically. I think it would be a huge gain.
https://www.hl7.org/fhir/observation.html
https://www.hl7.org/fhir/observation-example-body-height.htm...
The simple stuff like BMI, BSA, etc. have been calculated for a long time without anything fancy like an inference engine. The challenge is that the surface area of different calculations needed outstrip the supply of people who can interpret the clinical question, identify the source data and encode them. A better approach is needed and is what folks are working towards.
After 3 years of research, playing with datasets, extracting and cleaning data from EMR and from different machines, I not sure that the biggest problem with the so-called "AI" is the inequalities that it can induce ; it is rather, is it useful at all ? This is a little bit provocative so let me explain.
First, it took me a very long time to really, fully get that AI is not fundamentally different from a simple linear regression (infering a rule from data). More powerful, but definitely, no intelligence added. Most of my fellow doctor colleagues still think that AI is special, different, like magic; I also thought like this before learning to code.
Scores inferred from data were used in medicine from decades and fundamentally, nothing changed with the AI wave. I'm extremely impressed with the performances of GAN for image generation, and with what allows deep RL in controlled environments (which the human body is not): however, I can't see any applications of those technologies in medicine.
Ok, Deep Learning allows to analyze images with a great level of performances. However, at the end of the day, nobody (informed) wants an AI diagnosis and the doctor will read the images. He will maybe have a pre-completed report : wow, incredible. We are very far from the disappearance of radiologists that Geoffrey Hinton took for granted a few years ago.
At this time, a team published in Nature a paper about a DL algorithm which could diagnose melanomas better than dermatologists, using a picture. Unfortunately, no real-life application. Why? Because when you suspect a melanoma, if you have any doubt, you won't take a chance: you will take a biopsy. What is the interest of guessing the result of the biopsy that you will do anyway, because if you guessed wrong, the patient dies? No interest.
I also realized that it is extremely difficult, if not impossible, to use data from EMR out of the box. Medical data is dirty, intrinsically, because humans are complex things that do not fit easily in little boxes. Hence, if you want quality data, you have to think your data collection in advance, and motivate all your fellow doctors to check the little boxes correctly. For many years (we are talking about big data, no ?) Of course, there is some exceptions, but most of the time the data cleaning process is extremely hard to perform (however, if a dedicated team of people with medical knowledge is concentrated on this work, things could be different. I had to clean the data myself).
I'll finish with the most ironic part : I dedicated a few years of my life to a topic where both optics and prediction from data are involved (intraocular lens calculation in cataract surgery). I tried a great deal of ML approaches, only to find recently that by better accounting for the optical specificities of the problem I was trying to solve, I obtained excellent results, better than with ML, even with a dumb multiple regression. Ouch. The lesson is : physics beats AI.
I would be happy to be challenged on this topic.