Re:Production AI systems are hard
by haldujai on Monday May 29, 2023 @07:14PM
(#36119239)
You're missing the point, it takes me < 5 seconds to clear the adrenals. The example is intended to illustrate that there is no feature to extract that would make a model BETTER than a human for the things that we care about (rare and challenging diagnoses).
No one is arguing that ML can't segment and measure a structure, this is the lowest hanging fruit. ML can't diagnose an adrenocortical carcinoma (an example of a rare disease) because medicine doesn't know how to.
> In other words, we extract a single feature, and apply a single nonlinear activation function to that feature to decide whether or not to activate the 'treat' signal.
Now do this for the > 1000 other possible diagnoses on a CT abdomen, and have it be as fast as a human with equal or better ROC curve in under 5 seconds and cheaper than the $70 a radiologist bills for this. Unless you can eliminate having someone like me read this scan a ML model to measure the adrenal glands is worth $0.
I'm aware of the literature in this space. Your hypothetical is not novel and has been attempted. Everything from logistic regression to GANs and ViLMs has been tried and SOTA can barely beat human performance on an x-ray in idealized situations.
A genuine offer, my group has funding to support a FTE on this project so if you (or any reader) think you can solve this on a test set please reach out, my e-mail is in my profile.
> Look at how LLMs are able to do stuff like write code that has comments written in pirate speak.
You mean the things that hallucinate ~40% of the time? LLMs aren't even ready for a production CRUD app let alone human life.
> ML models looking at diagnoses with small training sets that are largely in obsolete scan formats could stil
It's not obsolete. It's a completely different image type. This is akin to saying a ML model trained on black and white sketches can paint the Mona Lisa in 3D.
> all the anatomical knowledge.
A misunderstanding of the problem. The anatomy is easy. The pathology is updated every 1-5 years so there is no historical dataset.
No one is arguing that ML can't segment and measure a structure, this is the lowest hanging fruit. ML can't diagnose an adrenocortical carcinoma (an example of a rare disease) because medicine doesn't know how to.
> In other words, we extract a single feature, and apply a single nonlinear activation function to that feature to decide whether or not to activate the 'treat' signal.
Now do this for the > 1000 other possible diagnoses on a CT abdomen, and have it be as fast as a human with equal or better ROC curve in under 5 seconds and cheaper than the $70 a radiologist bills for this. Unless you can eliminate having someone like me read this scan a ML model to measure the adrenal glands is worth $0.
I'm aware of the literature in this space. Your hypothetical is not novel and has been attempted. Everything from logistic regression to GANs and ViLMs has been tried and SOTA can barely beat human performance on an x-ray in idealized situations.
A genuine offer, my group has funding to support a FTE on this project so if you (or any reader) think you can solve this on a test set please reach out, my e-mail is in my profile.
> Look at how LLMs are able to do stuff like write code that has comments written in pirate speak.
You mean the things that hallucinate ~40% of the time? LLMs aren't even ready for a production CRUD app let alone human life.
> ML models looking at diagnoses with small training sets that are largely in obsolete scan formats could stil
It's not obsolete. It's a completely different image type. This is akin to saying a ML model trained on black and white sketches can paint the Mona Lisa in 3D.
> all the anatomical knowledge.
A misunderstanding of the problem. The anatomy is easy. The pathology is updated every 1-5 years so there is no historical dataset.