News for hackers. Stuff that matters.
Re:GenAI and erroneous medical references by larsiusprime on Saturday February 24, 2024 @07:01PM (#39496268)
> Unfortunately, very little evidence exists about the ability of LLMs to substantiate claims. In a new preprint study, we develop an approach to verify how well LLMs are able to cite medical references and whether these references actually support the claims generated by the models.

Is there a corresponding control group for how well an average doctor is able to cite medical references and whether these references actually support the claims generated by the doctors?

Re:GenAI and erroneous medical references by jncfhnb on Saturday February 24, 2024 @07:09PM (#39496320)
Doctors don’t cite claims on the spot
Re:GenAI and erroneous medical references by MauranKilom on Saturday February 24, 2024 @07:12PM (#39496344)
A fair comparison (on a technical level) to GPT-4 RAG would be a doctor in a relevant field who also has internet access. I think this would be indeed interesting to compare to assess the resulting quality of care, so to speak!

(The other models being only partially able to source good references is unsurprising/"unfair" on a technical level, but that's not relevant for assessing their safety.)

Re:GenAI and erroneous medical references by larsiusprime on Saturday February 24, 2024 @07:37PM (#39496490)
They often give clinical recommendations and prescriptions, usually after a quick skim of the medical history and a rushed five minute conversation with the patient. It would be nice to know would be of how many of these typical in-office recommendations wind up being actually backed by the current state of the research, whether a citation is given in the visit or not.
Re:GenAI and erroneous medical references by salad-tycoon on Saturday February 24, 2024 @08:48PM (#39496857)
It’s more like this, there is an algorithm / flow chart. You follow it down and do the next step in the process.

You learn the reason why and then you just follow whatever the latest update to the algorithm is. Doctors are then graded by administrators as to how well they follow that flow chart. Have you done this or that screening, is BP well controlled, etc. Medicare sets many of the standards, you look up the USPTF guidelines for some more idea of what your doctor should be doing. So yeah. Kinda like factory work. “Pt male 70 wakes up urinate - does he have signs of infection > yes/ no> proceed to no > next step do this test > prescribe this med > re evaluate in 2-4 weeks. “

Re:GenAI and erroneous medical references by __loam on Saturday February 24, 2024 @11:26PM (#39497645)
Why would we use proven processes involving experts with over a decade of education like this when we could use an LLM thrown together by a software developer who may not have even taken a biology course?
Re:GenAI and erroneous medical references by razodactyl on Sunday February 25, 2024 @03:25AM (#39498769)
Spot on
Re:GenAI and erroneous medical references by esoleyman on Saturday February 24, 2024 @07:15PM (#39496359)
Give me a few minutes and I can pull up any number of medical studies or references to back up my claims.

I don’t have them memorized to the actual URL but I have kept up to date with the latest studies and summaries that pertain to my field and my patients.

Re:GenAI and erroneous medical references by numpad0 on Saturday February 24, 2024 @07:25PM (#39496421)
Some engineers can cite RFCs and ISO standards on the spot sometimes, probably same for doctors
Re:GenAI and erroneous medical references by SkyPuncher on Saturday February 24, 2024 @07:19PM (#39496384)
Neither do most professionals- unless they’re doing a prepared segment.

The difference is LLMs can’t back their claims.

Re:GenAI and erroneous medical references by paulddraper on Saturday February 24, 2024 @11:00PM (#39497490)
No, but studies have found that 250k+ deaths per year in the US are due to medical errors. [1]

[1] https://www.hopkinsmedicine.org/news/media/releases/study_su...

Re:GenAI and erroneous medical references by __loam on Saturday February 24, 2024 @11:19PM (#39497594)
Doctors also go to school to be doctors for 12 extra years. It turns out that medicine is hard. Not really a good reason to turn to an LLM that will just confidently make things up.
Re:GenAI and erroneous medical references by paulddraper on Saturday February 24, 2024 @11:29PM (#39497664)
Medicine being hard is an excellent reason to use AI.
Re:GenAI and erroneous medical references by larsiusprime on Saturday February 24, 2024 @11:47PM (#39497746)
I’m aware, I come from a medical family. You haven’t answered the question.
Re:GenAI and erroneous medical references by esoleyman on Sunday February 25, 2024 @05:59AM (#39499746)
Martin Makary’s study and the previous IOM one are based on faulty statistics used. The number is extrapolated from a small population to a larger one.

I haven’t paid it any attention because of this problem. GIGO.

https://www.sciencealert.com/no-500-people-don-t-die-in-the-...

Re:GenAI and erroneous medical references by epcoa on Sunday February 25, 2024 @08:01AM (#39500451)
Yes perhaps read your own link:

“ The researchers caution that most of medical errors aren’t due to inherently bad doctors, and that reporting these errors shouldn’t be addressed by punishment or legal action. Rather, they say, most errors represent systemic problems, including poorly coordinated care, fragmented insurance networks, the absence or underuse of safety nets, and other protocols, in addition to unwarranted variation in physician practice patterns that lack accountability.”

An LLM is not going to address any of that. You are misinformed implying a significant majority of medical system errors are due to misdiagnosis.

Re:GenAI and erroneous medical references by paulddraper on Sunday February 25, 2024 @10:15AM (#39501400)
The availability, computerized nature, and economics of LLMs can certainly impact all of those factors.
Re:GenAI and erroneous medical references by epcoa on Sunday February 25, 2024 @02:00PM (#39503652)
Replace LLMs with smart phones or desktop calculators it doesn’t really change that statement except the tense and though not false it isn’t a particularly profound observation. I am skeptical of how much lifting “impact” is doing here.
Re:GenAI and erroneous medical references by melagonster on Sunday February 25, 2024 @12:29AM (#39497963)
I sure most of treatment are from textbooks.
Re:GenAI and erroneous medical references by extragood on Sunday February 25, 2024 @12:47AM (#39498032)
This has been my major concern with the currently available LLMs.

You can know what the input is, you can know the output, you may even be aware what it's been trained on, but none of the output is ever cited. Unless you are already familiar with the topic, you cannot confidently distinguish between fact and what sounds reasonable and is accepted as fact.

Re:GenAI and erroneous medical references by mateo1 on Sunday February 25, 2024 @04:33AM (#39499243)
Most doctors will be able to turn around, pick the corresponding textbook from their library and show you where they learned something. Or point you to an actual clinical case they had to handle. If they had to, somehow.