Re:OpenAI O3-Mini
by johnmaguire on Friday January 31, 2025 @05:03PM
(#42892726)
> If 10% of people just click based on how fast the response was
Couldn't this be considered a form of preference?
Whether it's the type of preference OpenAI was testing for, or the type of preference you care about, is another matter.
Sure, it could be, you can define "preference" as basically anything, but it just loses its meaning if you do that. I think most people would think "56% prefer this product" means "when well-informed, 56% of users would rather have this product than the other".
Re:OpenAI O3-Mini
by throwaway287391 on Friday January 31, 2025 @05:05PM
(#42892752)
Yes, I am assuming they evaluated the models in good faith, understand how to design a basic user study, and therefore when they ran a study intended to compare the response quality between two different models, they showed the raters both fully-formed responses at the same time, regardless of the actual latency of each model.
I would recommend you read the comment that started this thread then, because that's the context we're talking about: https://news.ycombinator.com/item?id=42891294
Re:OpenAI O3-Mini
by throwaway287391 on Friday January 31, 2025 @05:17PM
(#42892921)
I did read that comment. I don't think that person is saying they were part of the study that OpenAI used to evaluate the models. They would probably know if they had gotten paid to evaluate LLM responses.
But I'm glad you pointed that out, I now suspect that is responsible for a large part of the disagreement between "huh? a statistically significant blind evaluation is a statistically significant blind evaluation" vs "oh, this was obviously a terrible study" repliers is due to different interpretations of that post. Thanks. I genuinely didn't consider the alternative interpretation before.
If 10% of people just click based on how fast the response was because they don't want to read both outputs, your p-value for the latter hypothesis will be atrocious, no matter how large the sample is.