I would argue that GPT3 is way better at the last task, while also baking it with with a dozen or so made-up previous rulings that nobody will bother to check.
Re:GPT Takes the Bar Exam
by flanked-evergl on Monday January 09, 2023 @07:47AM
(#34309721)
> One of the few things I can currently see GPT do well is helping find legal precedence.
GPT is really not good at finding anything, and actually it can't find anything. It is not designed to do that. It is designed to make things up.
The difference may be subtle, but it is quite important. GPT makes up APIs, citations, tools, papers and rules. It makes up anything really. What it makes up is incredibly plausible, as it was trained to be as plausible as possible, to the point of being correct most of the time, but it still is just making things up.
If you use it with some different expectation you will be disappointed.
Re:GPT Takes the Bar Exam
by vasilipupkin on Monday January 09, 2023 @08:05AM
(#34309871)
I keep seeing comments along these lines. It doesn’t “make things up”. It outputs text under the objective function of maximizing probability of the token being outputted as “making sense” on some criteria. So, actually, it is biased towards not making things up. Occasionally, it hallucinates. I suspect we’ll see subsequent versions hallucinate less and less.
Re:GPT Takes the Bar Exam
by bryankaplan on Monday January 09, 2023 @08:41AM
(#34310170)
So what you're saying is it makes things up.
Re:GPT Takes the Bar Exam
by vasilipupkin on Monday January 09, 2023 @08:49PM
(#34319285)
It occasionally does. But it’s biased against it. It doesn’t make things up as it’s default behavior.
You still need a human to make the case, present the arguments and adapt every to the current situation, but there's no reason to have human search through thousands of cases and write summaries, not when an AI can do it in an instance.