News for hackers. Stuff that matters.
Re:Re-Evaluating GPT-4's Bar Exam Performance by Zenzero on Monday June 03, 2024 @10:42AM (#40563097)
Anything deeper than surface level in medicine.

Try getting it to properly select crystalloids with proper additives for a patient with a given history and lab results and watch in horror as it confidently gives instructions that would kill the patient.

What is even more irritating is that I had gpt4 debate me on things that it was completely wrong about and it was only when I responded with a stern rebuke that it hit me with the usual "Apologies for the misunderstanding..."

Re:Re-Evaluating GPT-4's Bar Exam Performance by mensetmanusman on Monday June 03, 2024 @01:11PM (#40564765)
LLMs are not good at answering expert level questions at the forefront of human knowledge.
Re:Re-Evaluating GPT-4's Bar Exam Performance by Zenzero on Monday June 03, 2024 @03:19PM (#40566448)
Unfortunately it would be considered basic medicine in this case.
Re:Re-Evaluating GPT-4's Bar Exam Performance by mensetmanusman on Monday June 03, 2024 @04:08PM (#40566948)
Is it basic but not documented? Basic to me means the first google search result is generally correct.
Re:Re-Evaluating GPT-4's Bar Exam Performance by Zenzero on Monday June 03, 2024 @06:46PM (#40568525)
That's not how medicine operates.

Medical problems are highly contextual, so you are not going to get much valuable information at the level of what a doctor is thinking from the first page of Google. That doesn't mean it isn't a simple within our area of expertise.

Re:Re-Evaluating GPT-4's Bar Exam Performance by mensetmanusman on Wednesday June 05, 2024 @08:17AM (#40583959)
In my area of expertise, a well formulated google search can result in a page 1 full of academic articles on the general topic, but there isn’t necessarily consensus. This might be a case of the curse of knowledge :)
Re:Re-Evaluating GPT-4's Bar Exam Performance by mordymoop on Monday June 03, 2024 @04:04PM (#40566918)
To be fair, I have not found MDs to be particularly reliable for answering basic questions about medicine either.
Re:Re-Evaluating GPT-4's Bar Exam Performance by Zenzero on Monday June 03, 2024 @06:47PM (#40568536)
OK. I can't speak for what you've experienced. I can only offer what I see from LLMs given what I know.