News for hackers. Stuff that matters.
Re:Re-Evaluating GPT-4's Bar Exam Performance by radford-neal on Saturday June 01, 2024 @10:11AM (#40545892)
A basic problem with evaluations like these is that the test is designed to discriminate between humans who would make good lawyers and humans who would not make good lawyers. The test is not necessarily any good at telling whether a non-human would make a good lawyer, since it will not test anything that pretty much all humans know, but non-humans may not.

For example, I doubt that it asks whether, for a person of average wealth and income, a $1000 fine is a more or less severe punishment than a month in jail.

Re:Re-Evaluating GPT-4's Bar Exam Performance by anon373839 on Sunday June 02, 2024 @12:29AM (#40551354)
Honestly, this is giving the bar exam (and GPT-4) too much credit. The bar tests memorization because it's challenging for humans and easy to score objectively. But memorization isn't that important in legal practice; analysis is. LLMs are superhuman at memorization but terrible at analysis.
Re:Re-Evaluating GPT-4's Bar Exam Performance by lazide on Sunday June 02, 2024 @03:19PM (#40556570)
Eh, also in legal practice there are key skills like selecting the best billable clients, covering your ass, building a reputation, choosing the right market segment, etc. which I’d also argue LLMs suck at.
Re:Re-Evaluating GPT-4's Bar Exam Performance by gadflyinyoureye on Sunday June 02, 2024 @03:48PM (#40556757)
I don’t know. There was some talk this weekend about CEOs being replaced by AI. Given the overlap in skill, I’d say there is a distinct possibility an LLM could do that. https://www.msn.com/en-us/money/companies/ceos-could-easily-...
Re:Re-Evaluating GPT-4's Bar Exam Performance by lazide on Sunday June 02, 2024 @03:54PM (#40556792)
Bwahaha. This is like the ‘everything can be a directed graph db’, ‘everything should be a micro service’, etc. fads.

No one who has been a CEO, or frankly even worked closely with one, would think this could be even remotely close to possible. Or desirable if it was.

But that is probably 1% or less of the population eh?

Re:Re-Evaluating GPT-4's Bar Exam Performance by EGreg on Sunday June 02, 2024 @05:25PM (#40557300)
Re:Re-Evaluating GPT-4's Bar Exam Performance by lazide on Sunday June 02, 2024 @05:29PM (#40557316)
Bwaha. Funny the company named as doing so doesn’t mention it on their actual management team [http://www.netdragon.com/about/management-team.shtml], listing an actual human CEO instead.

But it makes for a fun soundbite eh? Especially when the article claims it was in the past, and totally was awesome. Sucker born every minute.

Re:Re-Evaluating GPT-4's Bar Exam Performance by threeseed on Sunday June 02, 2024 @07:06PM (#40557839)
Phoebe Moore who that quote was attributed to has never been a CEO or even worked at a non-academic organisation.

So much of a what a CEO does is fostering culture, hiring people and setting a unique vision for the company.

Imagine thinking people would be inspired to work for a chatbot. Hilariously ridiculous.

Re:Re-Evaluating GPT-4's Bar Exam Performance by pojzon on Monday June 03, 2024 @04:41AM (#40560540)
If that chatbot had Steve Jobs voice ?

I dunno, I would probably prefer to work under that chatbot than my current CEO that only tries to squize as much as possible out of ppl already working for him.

Re:Re-Evaluating GPT-4's Bar Exam Performance by lazide on Monday June 03, 2024 @09:16AM (#40562333)
Like the chatbot wouldn’t squeeze you 10x harder.

At least a human CEO has to worry about being arrested or someone setting their house on fire.

Re:Re-Evaluating GPT-4's Bar Exam Performance by mrguyorama on Monday June 03, 2024 @03:00PM (#40566180)
Steve Jobs didn't even worry about cancer enough to save his life. Why the fuck do you think he would have an even remote understanding that squeezing people could result in consequences for him?
Re:Re-Evaluating GPT-4's Bar Exam Performance by lazide on Monday June 03, 2024 @04:47PM (#40567371)
I don’t think you’re making the point you think you’re making here.
Re:Re-Evaluating GPT-4's Bar Exam Performance by paulcole on Sunday June 02, 2024 @04:25PM (#40556981)
> which I’d also argue LLMs suck at

OK, I’ll bite. What’s your evidence for this argument?

Re:Re-Evaluating GPT-4's Bar Exam Performance by lazide on Sunday June 02, 2024 @04:31PM (#40557020)
Every bit of interaction I’ve ever had with an LLM. And all the research I’ve seen.

They’re plausible word sequence generators, not ‘planning for the future’ agents. Or market analyzers. Or character evaluators. Or anything else.

And they tend to be really ‘gullible’.

What evidence do you have they could do any of those things? (And not just generate plausible text at a prompt, but actually do those things)

Re:Re-Evaluating GPT-4's Bar Exam Performance by paulcole on Sunday June 02, 2024 @04:55PM (#40557139)
> What evidence do you have they could do any of those things?

Every bit of interaction I’ve ever had with an LLM.

Re:Re-Evaluating GPT-4's Bar Exam Performance by Terr_ on Monday June 03, 2024 @03:31AM (#40560211)
Yeah, I fear a lot of human exuberance (and thus investment) is riding on the questionable idea that a really good text-fragment-correlation specialist engine can usefully impersonate a generalist "thinking" AI without causing too much damage. ("LLM, which rocks are the best to eat?")

It gets worse: Some people are assuming an exceptional text-specialist model can also meta-impersonate a generalist model impersonating a specific and different kind of specialist! ("LLM, create a legal defense.")

Re:Re-Evaluating GPT-4's Bar Exam Performance by ethbr1 on Sunday June 02, 2024 @03:46PM (#40556747)
I've always drawn the link between skill in memorization and in analysis as:

- Memorization requires you to retain the details of a large amount of material

- The most time-efficient analysis uses instant-recall of relevant general themes to guide research

- Ergo, if someone can memorize and recall a large number of details, they can probably also recall relevant general themes, and therefore quickly perform quality analysis

(Side note: memorization also proves you actually read the material in the first place)

Re:Re-Evaluating GPT-4's Bar Exam Performance by Jensson on Sunday June 02, 2024 @04:45PM (#40557100)
Problem is the LLM memorized the countless examples you can find of old BAR questions using extreme amounts of compute at training time, they don't have that ability to digest a specific case due to both lack of data and it doesn't retrain for new questions.

A human that can digest the general law can also digest a special case, but that isn't true for an LLM.

Re:Re-Evaluating GPT-4's Bar Exam Performance by anon373839 on Sunday June 02, 2024 @06:53PM (#40557762)
I’m not sure why you’re being downvoted for this. I agree with you, fact recall is useful and necessary. If you have a larger and more tightly connected base of facts in your head, you can draw better connections.

And even though legal practice tends to be fairly slow and deliberative, there are settings (such as trial advocacy) where there is a real advantage to being able to cite a case or statute from memory.

All that said, I still maintain that it’s a poor way to compare humans with machines, for the same reason it would be poor to compare GPT-4 to a novelist on their tokens per second written.

Re:Re-Evaluating GPT-4's Bar Exam Performance by KennyBlanken on Sunday June 02, 2024 @04:04PM (#40556847)
You clearly don't know anything about the bar. One half of your score is split between 6 essay questions, and reviewing two cases to then follow instructions from a theoretical lead attorney.
Re:Re-Evaluating GPT-4's Bar Exam Performance by anon373839 on Sunday June 02, 2024 @06:35PM (#40557665)
I’m licensed in multiple states, including California.

The essay questions also test memorization. They don’t require any difficult analysis - just superficial issue-spotting and reciting the correct elements.

If the bar exam were not a memorization test, it would be open book!

Re:Re-Evaluating GPT-4's Bar Exam Performance by justinpombrio on Sunday June 02, 2024 @03:44PM (#40556737)
For a person of average wealth and income, is a $1000 fine is a more or less severe punishment than a month in jail? Be brief.

"For a person of average wealth and income, a $1000 fine is generally less severe than a month in jail. A month in jail entails loss of freedom, potential loss of employment, and social stigma, while a $1000 fine, though financially burdensome, does not affect one's freedom or ability to work" --ChatGPT 4o

Re:Re-Evaluating GPT-4's Bar Exam Performance by johnchristopher on Sunday June 02, 2024 @04:17PM (#40556925)
"potential loss of employment,"

Where is that coming from ? That's a very lawyery way to phrase things.

"potential ?" where I live I think people may max out their holidays and overtime (if lucky enough) and leave-without-pay but there would be a conversation with your employer to justify it and how to handle the workload.

In the USA, from what I read, it's more than likely that you would just be fired on the spot, right ?

edit: just googled a bit, where I live you must tell your employer why you will be absent if you go to jail but that can't be used to justify the breaking of the contract unless the reason for the incarceration is damaging to the company and... yeah, I am definitely not a lawyer :]

Re:Re-Evaluating GPT-4's Bar Exam Performance by ghaff on Sunday June 02, 2024 @04:22PM (#40556965)
Leave-without-pay normally requires some specific justification(s)/discussion. I've certainly given my manager advanced notice about any longer stretches of vacation and I've tried to do it with awareness of workloads (though for something planned months in advance that's not always possible) but I've pretty much never considered it as asking for permission or it being a negotiation. This is in the US.

ADDED: You're probably going to end up lying or at least being very vague "some family stuff to take care of" in this specific scenario but for one month that didn't trigger reporting to employer a lot of professionals could probably get off with it. In any case, the GPT answer seems totally correct for the parameters given.

Re:Re-Evaluating GPT-4's Bar Exam Performance by johnchristopher on Monday June 03, 2024 @02:12AM (#40559818)
> ADDED: You're probably going to end up lying or at least being very vague "some family stuff to take care of" in this specific scenario but for one month that didn't trigger reporting to employer a lot of professionals could probably get off with it. In any case, the GPT answer seems totally correct for the parameters given.

Where I live that is subject to cancellation of the work contact, you can't lie about why you are absent though the imprisonment can't be cause by itself for laying off.

Re:Re-Evaluating GPT-4's Bar Exam Performance by ghaff on Monday June 03, 2024 @10:22AM (#40562899)
Maybe you're talking about something else but for standard PTO/vacation that I'm owed it would seem absurd if I had to justify how I was going to spend my time off. It's none of your business.
Re:Re-Evaluating GPT-4's Bar Exam Performance by johnchristopher on Monday June 03, 2024 @11:01AM (#40563278)
Obviously not talking about standard PTO/vacation.

Where I live you must inform your employer unless you are lucky enough to have enough PTO/va. to spend your PTO/vacation in prison and hide it from your employer. But even then I don't think that will work out because there are administrative stuff related to social welfare you have to comply to and at some point it will be on your employer's radar anyway (why is that guy exempt from social welfare taxes for that specific month ? and why did the police asked me to confirm he was working here ? etc.).

BUT in practice, until recently, if imprisonment is less than 3 years then you won't spend a day in prison (unless you are deemed too dangerous). But then you have an electronic bracelet and other obligations that will at some point alert your employer.

I read now that prison time of any length will have to be spent in prison (no more less than 3 years or 2 years or 18 months pass). But for prison time less than 18 months you can have a bracelet on the first day.

All that to say I don't think it's manageable and possible to hide prison time from your employer, even if you have enough PTO/vac. days to cover for it.

Re:Re-Evaluating GPT-4's Bar Exam Performance by oconnor663 on Sunday June 02, 2024 @04:22PM (#40556964)
My wild guess is that it would depend a lot on how much your employer likes you and how they feel about the reason you're in jail.
Re:Re-Evaluating GPT-4's Bar Exam Performance by akira2501 on Sunday June 02, 2024 @04:50PM (#40557119)
Many jails have work release. They get you up at 6am, check you out of the jail, let you go to work, then expect you to check back into jail by 6pm.
Re:Re-Evaluating GPT-4's Bar Exam Performance by ghaff on Sunday June 02, 2024 @04:56PM (#40557143)
Which for many professional (and other jobs) probably would require a bunch of tap-dancing around your strict schedule if you were hiding the actual reason.
Re:Re-Evaluating GPT-4's Bar Exam Performance by johnchristopher on Sunday June 02, 2024 @04:56PM (#40557142)
Oh, that's really great ! Is that in the US ?
Re:Re-Evaluating GPT-4's Bar Exam Performance by Dayshine on Sunday June 02, 2024 @04:22PM (#40556967)
Self-employed or small company might not care.
Re:Re-Evaluating GPT-4's Bar Exam Performance by EGreg on Sunday June 02, 2024 @04:21PM (#40556958)
So the GPT is on a law exam and is using a very lawyer way to word things? I would say that's great!!
Re:Re-Evaluating GPT-4's Bar Exam Performance by Rinzler89 on Sunday June 02, 2024 @03:55PM (#40556798)
What does GPT consider being "average wealth and income". Statistics? Or biased weights from anecdotes he formed on the anecdotes he scraped off the internet on how wealthy people say the feel?

Would be cool to know how LLMs shape their opinions.

Re:Re-Evaluating GPT-4's Bar Exam Performance by LeoPanthera on Sunday June 02, 2024 @03:57PM (#40556811)
You can just ask it, you know.

GPT-4o:

“Average wealth and income” can vary significantly by region and context. However, in the United States, as a rough benchmark, the median household income is around $70,000 per year. Wealth, which includes assets such as savings, property, and investments minus debts, is harder to pinpoint but median net worth for U.S. households is approximately $100,000. These figures provide a general idea of what might be considered “average” in terms of wealth and income."

Re:Re-Evaluating GPT-4's Bar Exam Performance by Rinzler89 on Sunday June 02, 2024 @04:05PM (#40556859)
>You can just ask it, you know.

But my question will not be part of the context of that conversation.

Re:Re-Evaluating GPT-4's Bar Exam Performance by LeoPanthera on Sunday June 02, 2024 @04:08PM (#40556876)
Mine was. I asked it the first question, first.
Re:Re-Evaluating GPT-4's Bar Exam Performance by Rinzler89 on Sunday June 02, 2024 @07:48PM (#40558058)
The original conversation I mean, of the grandparent. Me starting an identical conversation will not guarantee an identical context.
Re:Re-Evaluating GPT-4's Bar Exam Performance by lambdaxyzw on Sunday June 02, 2024 @04:26PM (#40556987)
I like that it immediately assumed the US, even though nothing in your question suggested it. I love that all LLMs have a strong US centric bias.

Btw I'm not personally a lawyer, but I've heard that GPT is especially prone to mixing laws across the borders - for example you ask a law question in language X, and get a response that uses a law from a country Y - and it's extremally convincing doing that (unless you're a lawyer, I guess).

Re:Re-Evaluating GPT-4's Bar Exam Performance by fjdjshsh on Sunday June 02, 2024 @05:06PM (#40557190)
This is my experience with Hackernews. If the comment doesn't specify the country, it's an American talking about the USA
Re:Re-Evaluating GPT-4's Bar Exam Performance by LeoPanthera on Sunday June 02, 2024 @04:37PM (#40557057)
ChatGPT has user-customizable "instructions", and mine are set to tell it where I live. Any user can do the same, so that it will not make incorrect assumptions for you.
Re:Re-Evaluating GPT-4's Bar Exam Performance by d3m0t3p on Sunday June 02, 2024 @05:11PM (#40557232)
You might increase the probability of getting a correct answer for your region, but imo you decrease your awareness to allucination. Overall you can still get a wrong answer
Re:Re-Evaluating GPT-4's Bar Exam Performance by LordDragonfang on Sunday June 02, 2024 @04:35PM (#40557043)
I mean, to be fair, if you're speaking English to it, the most likely possibility is that you're inside the US:

https://en.wikipedia.org/wiki/List_of_countries_by_English-s...

I know there's a lot of complaints about things being US-centric, but the US is a very large country.

Re:Re-Evaluating GPT-4's Bar Exam Performance by ordersofmag on Sunday June 02, 2024 @06:28PM (#40557622)
Well, except the number of English speakers outside the US is much larger than inside the US (as per the wikipedia page you point to) by 5 to 1. Granted many folks are speaking it as their 2nd (or nth) language. But when you take into account the limited set of languages supported by ChatGPT one could reasonably assume English-speaking (typing) users of ChatGPT are from outside the U.S. as non-U.S. folks are in the majority of 'folks for whom english would be their first option when interacting with ChatGPT'. Even if you only count India, Nigeria and Pakistan.

Though of course OpenAI can tell (frequently, roughly) where folks are coming from geographically and could (does?) take that into account.

Re:Re-Evaluating GPT-4's Bar Exam Performance by duskwuff on Sunday June 02, 2024 @11:31PM (#40559089)
> but the US is a very large country.

Indeed - the US is a very large country, and consists of over 50 different jurisdictions, each with their own slightly different laws. An answer to a legal question which is correct in one state will often be subtly incorrect in another, and completely wrong in yet another.