News for hackers. Stuff that matters.
Re:Legal models hallucinate in 1 out of 6 (or more) benchmar... by basch on Friday May 31, 2024 @05:47PM (#40540654)
That's a round peg in a square hole. As ive seen them called elsewhere today, these "plausible text generators" can create a pseudo facsimile of reasoning, but they don't reason, and they don't fact check. Even when they use sources to build consensus, its more about volume than authoritativeness.
Re:Legal models hallucinate in 1 out of 6 (or more) benchmar... by theturtletalks on Friday May 31, 2024 @06:16PM (#40540897)
I was watching the show, 3 Body Problem, and there was a great scene where a guy tells a woman to double check another man’s work. Then goes to the man and tells him to triple check the woman’s work. MoE seems to work this way, but maybe we can leverage different models that have different randomness and maybe we can get to a more logical answer.

We have to start thinking about LLM hallucination differently. When it’s follows logic correctly and provides factual information, that is also a hallucination, but one that fits our flow of logic.

Re:Legal models hallucinate in 1 out of 6 (or more) benchmar... by dougabug on Friday May 31, 2024 @11:09PM (#40542564)
Sure, but if we label the text as “factually accurate” or “logically sound” (or “unsound”) etc., then we can presumably greatly increase the probability of producing text with targeted properties
Re:Legal models hallucinate in 1 out of 6 (or more) benchmar... by salawat on Saturday June 01, 2024 @10:12AM (#40545901)
What on Earth makes you think that training a model on all factual information is going to do a lick in terms of generating factual outputs?

At that point, clearly our only problem has been we've done it wrong all along by not training these things only on academic textbooks! That way we'll only probabilistically get true things out, right? /s