News for hackers. Stuff that matters.
Re:Good Machine Learning Practice for Medical Device Develop... by rapjr9 on Sunday November 20, 2022 @09:49PM (#33688108)
Good wishlist, but how do you measure risk in a trained ML algorithm? What does reliability and stability mean? How do you measure bias? How much accuracy in results is enough for a specific purpose? These are all still research questions. Requiring diversity in training data inputs helps, but it is no guarantee of a fair or correct outcome, and there will likely always be weird edge cases where the algorithms go off the rails. Do you need to include Zulu people, Inuit people, and Hawaiian people in large proproportions in your training data? Maybe you need to train a different algorithm for each comorbidity a patient might present with? And all the combinations of those comorbidities? Does your training data have to include all the environmental factors that might affect the outcome? "Personalized medicine" recognizes that all people are unique and different in medically important ways, yet machine learning has a basic assumption that all people are the same and you can just dump data from everyone into the training and the algorithm will figure it out...somehow. This is unproven and likely just plain wrong.
Re:Good Machine Learning Practice for Medical Device Develop... by 5440 on Sunday November 20, 2022 @11:04PM (#33688614)
I think those are generally good questions but the majority of submissions that FDA CDRH are reviewing are for imaging of ultrasound, MRI, XRAY etc, with a sprinkling of audio AI and maybe some individualized vaccines predictions at CDER.

These questions are generally addressed during the pre-sub for each FDA submission but I agree that the demographic information could be pretty infinite. I'm working on at least 5 of these AI submissions per week.

For example, lets say you have an AI submission for identifying cardiac ultrasound images. FDA will ask for very specific demographic information in the training and performance tests, as well as comorbidities (such as hypertension) for each training image. In addition, they will want at least three physicians to annotate the images. The training dataset is likely to contain at least 100-200K images.

Re:Good Machine Learning Practice for Medical Device Develop... by edge17 on Sunday November 20, 2022 @11:51PM (#33688920)
If three physicians are annotating 100-200k images, how much does that cost?
Re:Good Machine Learning Practice for Medical Device Develop... by snowpid on Monday November 21, 2022 @04:42AM (#33690704)
One startup from Hamburg that I know uses medical students for this, and from what I always hear, their training dataset is their most valuable asset.
Re:Good Machine Learning Practice for Medical Device Develop... by ware_am_i on Monday November 21, 2022 @12:54AM (#33689319)
It's cool to see another FDA AI Regulatory person on HN. Drop me a note if you are interested in connecting -> [email protected]
Re:Good Machine Learning Practice for Medical Device Develop... by s1artibartfast on Sunday November 20, 2022 @11:58PM (#33688961)
These are all good questions indeed, and ones that companies will have to figure out. If you aren't familiar with the medical space, there are already some methods for addressing these issues.

You could ask the same question about how a pharmaceutical agent interacts with a diverse set of patients

Re:Good Machine Learning Practice for Medical Device Develop... by ware_am_i on Monday November 21, 2022 @12:33AM (#33689166)
As a medical device company, establishing and documenting your approach to answer these questions can go a long way. More of a policy vs protocol approach.