News for hackers. Stuff that matters.
Re:Re-Evaluating GPT-4's Bar Exam Performance by mordymoop on Sunday June 02, 2024 @06:37PM (#40557672)
On what topics you understand well does GOT-4o or Claude Opus produce garbage?
Re:Re-Evaluating GPT-4's Bar Exam Performance by threeseed on Sunday June 02, 2024 @06:46PM (#40557720)
I do run into the issue where the longer the conversation goes the more inaccurate the information.

But a common situation is that with code generation it will fail to understand the context of where the code belongs and so it's a function that will compile but makes no sense.

Re:Re-Evaluating GPT-4's Bar Exam Performance by fnordpiglet on Sunday June 02, 2024 @06:51PM (#40557750)
Yeah. I often springboard into a new context by having the LLM compose the next prompt based on the discussion and restart the context. Remarkably effective if you ask it to incorporate “prompt engineering” terms from research.
Re:Re-Evaluating GPT-4's Bar Exam Performance by lupire on Monday June 03, 2024 @07:27AM (#40561609)
High school math problems.
Re:Re-Evaluating GPT-4's Bar Exam Performance by Zenzero on Monday June 03, 2024 @10:42AM (#40563097)
Anything deeper than surface level in medicine.

Try getting it to properly select crystalloids with proper additives for a patient with a given history and lab results and watch in horror as it confidently gives instructions that would kill the patient.

What is even more irritating is that I had gpt4 debate me on things that it was completely wrong about and it was only when I responded with a stern rebuke that it hit me with the usual "Apologies for the misunderstanding..."

Re:Re-Evaluating GPT-4's Bar Exam Performance by mensetmanusman on Monday June 03, 2024 @01:11PM (#40564765)
LLMs are not good at answering expert level questions at the forefront of human knowledge.
Re:Re-Evaluating GPT-4's Bar Exam Performance by Zenzero on Monday June 03, 2024 @03:19PM (#40566448)
Unfortunately it would be considered basic medicine in this case.
Re:Re-Evaluating GPT-4's Bar Exam Performance by mensetmanusman on Monday June 03, 2024 @04:08PM (#40566948)
Is it basic but not documented? Basic to me means the first google search result is generally correct.
Re:Re-Evaluating GPT-4's Bar Exam Performance by Zenzero on Monday June 03, 2024 @06:46PM (#40568525)
That's not how medicine operates.

Medical problems are highly contextual, so you are not going to get much valuable information at the level of what a doctor is thinking from the first page of Google. That doesn't mean it isn't a simple within our area of expertise.

Re:Re-Evaluating GPT-4's Bar Exam Performance by mensetmanusman on Wednesday June 05, 2024 @08:17AM (#40583959)
In my area of expertise, a well formulated google search can result in a page 1 full of academic articles on the general topic, but there isn’t necessarily consensus. This might be a case of the curse of knowledge :)
Re:Re-Evaluating GPT-4's Bar Exam Performance by mordymoop on Monday June 03, 2024 @04:04PM (#40566918)
To be fair, I have not found MDs to be particularly reliable for answering basic questions about medicine either.
Re:Re-Evaluating GPT-4's Bar Exam Performance by Zenzero on Monday June 03, 2024 @06:47PM (#40568536)
OK. I can't speak for what you've experienced. I can only offer what I see from LLMs given what I know.