News for hackers. Stuff that matters.
Re:OpenAI O3-Mini by dkjaudyeqooe on Friday January 31, 2025 @03:31PM (#42891590)
It's kind of strange that they gave that stat. Maybe they thought people would somehow think about "56% better" or something.

Because when you think about it, it really is quite damning. Minus statistical noise it's no better.

Re:OpenAI O3-Mini by Powdering7082 on Friday January 31, 2025 @03:58PM (#42891882)
That would be 12%, why would you assume that is eaten by statistical noise?
Re:OpenAI O3-Mini by senorrib on Friday January 31, 2025 @04:08PM (#42892021)
The OPs comment is probably a testament of that. With such a poorly designed A/B test I doubt this has a p-value of < 0.10.
Re:OpenAI O3-Mini by throwaway287391 on Friday January 31, 2025 @04:49PM (#42892564)
Erm, why not? A 0.56 result with n=1000 ratings is statistically significantly better than 0.5 with a p-value of 0.00001864, well beyond any standard statistical significance threshold I've ever heard of. I don't know how many ratings they collected but 1000 doesn't seem crazy at all. Assuming of course that raters are blind to which model is which and the order of the 2 responses is randomized with every rating -- or, is that what you meant by "poorly designed"? If so, where do they indicate they failed to randomize/blind the raters?
Re:OpenAI O3-Mini by godelski on Friday January 31, 2025 @05:48PM (#42893262)

  > If so, where do they indicate they failed to randomize/blind the raters?

  Win rate if user is under time constraint
This is hard to read tbh. Is it STEM? Non-STEM? If it is STEM then this shows there is a bias. If it is Non-STEM then this shows a bias. If it is a mix, well we can't know anything without understanding the split.

Note that Non-STEM is still within error. STEM is less than 2 sigma variance, so our confidence still shouldn't be that high.

Re:OpenAI O3-Mini by n2d4 on Friday January 31, 2025 @05:00PM (#42892685)
Because you're not testing "will a user click the left or right button" (for which asking a thousand users to click a button would be a pretty good estimation), you're testing "which response is preferred".

If 10% of people just click based on how fast the response was because they don't want to read both outputs, your p-value for the latter hypothesis will be atrocious, no matter how large the sample is.

Re:OpenAI O3-Mini by johnmaguire on Friday January 31, 2025 @05:03PM (#42892726)
> If 10% of people just click based on how fast the response was

Couldn't this be considered a form of preference?

Whether it's the type of preference OpenAI was testing for, or the type of preference you care about, is another matter.

Re:OpenAI O3-Mini by n2d4 on Friday January 31, 2025 @05:16PM (#42892907)
Sure, it could be, you can define "preference" as basically anything, but it just loses its meaning if you do that. I think most people would think "56% prefer this product" means "when well-informed, 56% of users would rather have this product than the other".
Re:OpenAI O3-Mini by throwaway287391 on Friday January 31, 2025 @05:05PM (#42892752)
Yes, I am assuming they evaluated the models in good faith, understand how to design a basic user study, and therefore when they ran a study intended to compare the response quality between two different models, they showed the raters both fully-formed responses at the same time, regardless of the actual latency of each model.
Re:OpenAI O3-Mini by n2d4 on Friday January 31, 2025 @05:14PM (#42892878)
I would recommend you read the comment that started this thread then, because that's the context we're talking about: https://news.ycombinator.com/item?id=42891294
Re:OpenAI O3-Mini by throwaway287391 on Friday January 31, 2025 @05:17PM (#42892921)
I did read that comment. I don't think that person is saying they were part of the study that OpenAI used to evaluate the models. They would probably know if they had gotten paid to evaluate LLM responses.

But I'm glad you pointed that out, I now suspect that is responsible for a large part of the disagreement between "huh? a statistically significant blind evaluation is a statistically significant blind evaluation" vs "oh, this was obviously a terrible study" repliers is due to different interpretations of that post. Thanks. I genuinely didn't consider the alternative interpretation before.

Re:OpenAI O3-Mini by aqme28 on Friday January 31, 2025 @04:21PM (#42892209)
They even include error bars. It doesn't seem to be statistical noise, but it's still not great.
Re:OpenAI O3-Mini by afro88 on Friday January 31, 2025 @04:14PM (#42892099)
Yeah. I immediately thought: I wonder if that 56% is in one or two categories and the rest are worse?
Re:OpenAI O3-Mini by rvnx on Friday January 31, 2025 @05:53PM (#42893323)
44% of the people prefers the existing model ?
Re:OpenAI O3-Mini by afro88 on Friday January 31, 2025 @07:57PM (#42894472)
Each question falls into a different category (ie math, coding, story writing etc). Typically models are better at some categories and worse at others. Saying "56% of people preferred responses from o3-mini" makes me wonder if those 56 are only from certain categories and the model isn't uniformly 56% preferred.
Re:OpenAI O3-Mini by KHRZ on Friday January 31, 2025 @06:46PM (#42893862)
With many people too lazy to read 2 walls of text, a lot of picks might be random.
Re:OpenAI O3-Mini by fsndz on Friday January 31, 2025 @03:49PM (#42891799)
exactly I was surprised as well
Re:OpenAI O3-Mini by cm2187 on Friday January 31, 2025 @05:06PM (#42892780)
And another way to rephrase it is that almost half of the users prefer the older model, which is terrible PR.
Re:OpenAI O3-Mini by tgsovlerkhgsel on Friday January 31, 2025 @05:50PM (#42893281)
Not if the goal is to claim that the models deliver comparable quality, but with the new one excelling at something else (here: inferrence cost).
Re:OpenAI O3-Mini by swyx on Saturday February 01, 2025 @05:28AM (#42897375)
its mini to mini, its the same cost
Re:OpenAI O3-Mini by kettleballroll on Friday January 31, 2025 @06:18PM (#42893570)
Typically in these tests you have three options "A is better", "B is better" or "they're equal/can't decide". So if 56% prefer O3 Mini, it's likely that way less than half prefer O1.also, the way I understand it, they're comparing a mini model with a large one.
Re:OpenAI O3-Mini by directevolve on Friday January 31, 2025 @06:22PM (#42893621)
If you use ChatGPT, it sometimes gives you two versions of its response, and you have to choose one or the other if you want to continue prompting. Sure, not picking a response might be a third category. But if that's how they were approaching the analysis, they could have put out a more favorable-looking stat.
Re:OpenAI O3-Mini by ignoramous on Friday January 31, 2025 @06:34PM (#42893756)
> If you use ChatGPT, it sometimes gives you two versions

Does no one else hate it when this happens (especially when on a handheld device)?

Re:OpenAI O3-Mini by m3kw9 on Friday January 31, 2025 @06:35PM (#42893767)
It’s 3x cheaper and faster