Because when you think about it, it really is quite damning. Minus statistical noise it's no better.
> If so, where do they indicate they failed to randomize/blind the raters?
Win rate if user is under time constraint
This is hard to read tbh. Is it STEM? Non-STEM? If it is STEM then this shows there is a bias. If it is Non-STEM then this shows a bias. If it is a mix, well we can't know anything without understanding the split.Note that Non-STEM is still within error. STEM is less than 2 sigma variance, so our confidence still shouldn't be that high.
If 10% of people just click based on how fast the response was because they don't want to read both outputs, your p-value for the latter hypothesis will be atrocious, no matter how large the sample is.
Couldn't this be considered a form of preference?
Whether it's the type of preference OpenAI was testing for, or the type of preference you care about, is another matter.
But I'm glad you pointed that out, I now suspect that is responsible for a large part of the disagreement between "huh? a statistically significant blind evaluation is a statistically significant blind evaluation" vs "oh, this was obviously a terrible study" repliers is due to different interpretations of that post. Thanks. I genuinely didn't consider the alternative interpretation before.
Does no one else hate it when this happens (especially when on a handheld device)?
Which is why you randomize the order. You aren’t a tester.
56% vs 44% may not be noise. That’s why we have p values. It depends on sample size.
I would not consider purposely giving inaccurate feedback for this reason alone.
I want a single source model that's grounded in base truth. I'll let the model know how to structure it in my prompt.
I like precision of language, so maybe just have a system prompt that says "use precise language (ex: no symbolism of any kind)"
Wasting time on an anti pattern is not value nor is it trying to outguess the way that selection mechanism is used.
Didn’t bother reading either of them, just selected one and went on with my day.
If it were me I would have set up a “hey do you mind if we give you two results and you can pick your favorite?” prompt to weed out people like me.
We find what appears to be the easiest combination "Fuck off, go away" buttons and use them without a moment of actual consideration.
(This doesn't mean that it's actually the easiest method.)
1) Coming off as a jerk, and from a new account is a bad look
2) "Literally the opposite of a coin flip" would probably be either 0% or 100%
3) Your reasoning doesn't stand up without further info; it entirely depends on the sample size. I could have 5 coin flips all come up heads, but over thousands or millions it averages to 50%. 56% on a small sample size is absolutely within margin of error/noise. 56% on a MASSIVE sample size is _statistically_ significant, but isn't even still that much to brag about for something that I feel like they probably intended to be a big step forward.
1. The message was net-upvoted. Whether there are downvotes in there I can't tell, but the final karma is positive. A similarly spirited message of mine in the same thread was quite well receive as well.
2. I can't see how my message would come across as a jerk? I wrote 2 simple sentences, not using any offensive language, stating a mere fact of statistics. Is that being jerk? And a long-winded berating of a new member of the community isn't?
3. A coin flip is 50%. Anything else is not, once you have a certain sample size. So, this was not. That was my statement. I don't know why you are building a strawman of 5 coin flips. 56% vs 44% is a margin of 12%, as I stated, and with a huge sample size, which they had, that's massive in a space where the returns are deep in "diminishing" territory.
We're talking about the most cost-efficient model, the competition here is on price, not on a 12% incremental performance (which would make sense for the high end model).
To my knowledge deepseek is the cheaper service which is what matters on the low-end (unless the increase in performance was in such magnitude that the extra-charge would be worth the money).
I personally chose for price on a low-cost model (unless the improvement is to significant that it justifies the higher price).
The interface wastes so much screen real estate already and the answers are usually overly verbose unless I've given explicit instructions on how to answer.
"Evaluations by expert testers showed that o3-mini produces more accurate and clearer answers, with stronger reasoning abilities, than OpenAI o1-mini. Testers preferred o3-mini's responses to o1-mini 56% of the time and observed a 39% reduction in major errors on difficult real-world questions. W"
1) sometimes I wanted the short answer, and so even though the long answer is better I picked the short one.
2) sometimes both contain code that is different enough that I am inclined to go with the one that is more similar to what I already had, even if the other approach seems a bit more solid.
3) Sometimes one will have less detail but more big picture awareness and the other will have excellent detail but miss some overarching point that is valuable. Depending on my mood I sometimes choose but it is annoying to have to do so because I am not allowed to say why I made the choice.
The area of human training methodology seems to be a big part of what got Deepseek's model so strong. I read the explanation of the test results as an acknowledgement by OpenAI of some weaknesses in its human feedback paradigm.
IMO the way it should work is that the thumbs up or down should be read in context by a reasoning being and a more in-depth training case should be developed that helps future models learn whatever insight the feedback should have triggered.
Feedback that A is better or worse than B is definitely not (in my view) sufficient except in cases where a response is a total dud. Usually the responses have different strengths and weaknesses and it's pretty subjective which one is better.
I have something like “always be terse and blunt with your answers.”
I'm not going to say there's nothing substantive about o3 vs. o1, but I absolutely do not put it past Sam Altman to juice the stats every chance he gets.
I hope by this they don't mean me, when I'm asked 'which of these two responses do you prefer'.
They're both 2,000 words, and I asked a question because I have something to do. I'm not reading them both; I'm usually just selecting the one that answered first.
That prompt is pointless. Perhaps as evidenced by the essentially 50% response rate: it's a coin-flip.