News for hackers. Stuff that matters.
Re:Prostate cancer can be precisely diagnosed using a urine ... by brundolf on Thursday January 21, 2021 @11:34AM (#25860722)
How do clinical trials work for ML? Since the system is largely a black-box, I imagine that the slightest alteration (training on more data, tweaking any parameters) would require totally re-doing a full gamut of trials, wouldn't it?
Re:Prostate cancer can be precisely diagnosed using a urine ... by 5440 on Thursday January 21, 2021 @12:19PM (#25861357)
I'm a regulatory consultant for AI software and submit several AI/ML FDA submissions per week. Overall, I've submitted 100+ AI/ML submissions to date. Generally, FDA has been really focused on locked algorithms even though there has been some guidance lately stating the the new Good Machine Learning Practice (GMLP) will provide some allowance for iterations without submitting new 510ks. I will say that a number of companies are slightly tuning their algorithms without FDA resubmission within the confines of the FDA Guidance "https://www.fda.gov/regulatory-information/search-fda-guidan...". That said, most companies most companies are just leaving their specifications at sensitivity/specificity of 80% which gives them some leeway to improve above those limits. In truth most companies are above 97%-99%. While most of the submissions I work on are MRI/CT image related, I'm starting to get a lot more in the predictive space centered around ovarian (CA125), breast or PSA cancer scores. Lately pathology AI is escalating rapidly.
Re:Prostate cancer can be precisely diagnosed using a urine ... by brundolf on Thursday January 21, 2021 @01:21PM (#25862121)
Thanks for the informed and informative answer! I love how common it is on HN to hear from someone with first-hand knowledge of a topic
Re:Prostate cancer can be precisely diagnosed using a urine ... by Kuinox on Thursday January 21, 2021 @01:03PM (#25861921)
A lot of thing are blackbox. There also things that we think we understand, but we do not, thats why we have methodologies like double blind testing to assert that the blackbox is indeed working, and it's behavior is stable enough to be used as a tool.
Re:Prostate cancer can be precisely diagnosed using a urine ... by swsieber on Thursday January 21, 2021 @11:58AM (#25861089)
I'm not sure why you would need to re-do a full gamut of trials for anything except deciding that the current type of data collected was insufficient.

Presumably you record all data, and just split it into test and train sets, and you're off to the races.

Re:Prostate cancer can be precisely diagnosed using a urine ... by brundolf on Thursday January 21, 2021 @12:16PM (#25861323)
I guess it makes sense that unlike for actual treatments, for diagnostic systems you could just "re-use" the test data from past trials and not have to gather a whole new dataset