News for hackers. Stuff that matters.
Re:Re-Evaluating GPT-4's Bar Exam Performance by WhitneyLand on Sunday June 02, 2024 @07:07PM (#40557847)
I suspect by garbage you mean not perfect.

To be more precise can you please give a topic you know well and your % guess how often the answers are wrong on the topic?

Re:Re-Evaluating GPT-4's Bar Exam Performance by lanstin on Sunday June 02, 2024 @07:53PM (#40558088)
Functional linear analysis - it has tendency to produce a proof for unprovable statements; the proofs will be logically argued and well structured and step 8 will have a statement that is obvious nonsense even to a beginning student, me. The professor on the other hand will ask why I'm trying to prove the false statement and expertly help me find my logic error.
Re:Re-Evaluating GPT-4's Bar Exam Performance by WhitneyLand on Sunday June 02, 2024 @08:39PM (#40558295)
Specifics like this make it much easier to agree on LLM capabilities, thank you.

Automatic proof generation is a massive open problem in all of computer science and not close to be solved. It’s true LLMs aren’t great at it and more is required for example as with the geometry system Deepmind progresses on.

On the other hand they can be very useful to explain concepts and allow interactive questioning to drill down and help build understanding of complex mathematical concepts, all during a morning commute via the voice interface.

Re:Re-Evaluating GPT-4's Bar Exam Performance by lupire on Monday June 03, 2024 @07:27AM (#40561606)
How do yo debug its hallucination misinformation via voice interface while you commute?
Re:Re-Evaluating GPT-4's Bar Exam Performance by fnordpiglet on Monday June 03, 2024 @03:20PM (#40566471)
I just use my memory and verify later. Unlike a LLM I have persistent long term durable storage of knowledge. Typically I can pretty easily pick out a hallucination though because there’s often a very clear inconsistency or logical leap that is non sense.
Re:Re-Evaluating GPT-4's Bar Exam Performance by rcxdude on Sunday June 02, 2024 @07:17PM (#40557897)
I would take their meaning as 'contains enough errors to not be useful', which doesn't need a very high percentage of wrong answers.
Re:Re-Evaluating GPT-4's Bar Exam Performance by lazide on Sunday June 02, 2024 @08:30PM (#40558256)
Even better, looks right, might even compile, but will be doing the subtly (or obviously) wrong thing.
Re:Re-Evaluating GPT-4's Bar Exam Performance by supriyo-biswas on Monday June 03, 2024 @12:04AM (#40559226)
I’m not the parent, but depending on the context, GPT-4 will often make up functions that then end up requiring research and correction; in other cases like once when I asked it to show me an example of a class of X86 assembly instructions, it just added a label and skipped the actual instruction and implementation!

Yesterday I was looking for some help on an issue with the unshare command; it repeatedly made bad assumptions about the nature of the error even I provided it with the full error message and one could already guess the initial cause by looking at that.

I guess such errors can be frighteningly common once you get outside of typical web development.