News for hackers. Stuff that matters.
Re:GPT-5.2 by minadotcom on Thursday December 11, 2025 @01:29PM (#46235074)
They used to compare to competing models from Anthropic, Google DeepMind, DeepSeek, etc. Seems that now they only compare to their own models. Does this mean that the GPT-series is performing worse than its competitors (given the "code red" at OpenAI)?
Re:GPT-5.2 by Tiberium on Thursday December 11, 2025 @01:35PM (#46235145)
Re:GPT-5.2 by enlyth on Thursday December 11, 2025 @02:22PM (#46235919)
This looks cherry-picked, for example Claude Opus had a higher score on SWE-Bench Verified so they conveniently left it out, also GDPval is literally a benchmark made by OpenAI
Re:GPT-5.2 by minadotcom on Thursday December 11, 2025 @04:24PM (#46237361)
agreed.
Re:GPT-5.2 by tobias2014 on Thursday December 11, 2025 @08:41PM (#46239897)
And who believes that the difference between 91.9% and 92.4% is significant in these benchmarks? Clearly these have margins of error that are swept under the rug.
Re:GPT-5.2 by whimsicalism on Thursday December 11, 2025 @05:27PM (#46238146)
uh oh, where did SWE bench go :D
Re:GPT-5.2 by whimsicalism on Thursday December 11, 2025 @09:11PM (#46240109)
maybe they will release with gpt-5.2-codex
Re:GPT-5.2 by sergdigon on Friday December 12, 2025 @02:20AM (#46241683)
The fact that the post is comparing their reasoning model against gemini 3 pro (the "non reasoning" model) and not gemini 3 pro deep think (the reasoning one) is quite nasty. If you compare GPT5.2 thinking to gemini 3 pro deep think, the scores are quite similar (sometimes one is better sometimes the other one is)
Re:GPT-5.2 by tabletcorry on Thursday December 11, 2025 @01:32PM (#46235110)
The matrix required for a fair comparison is getting too complicated, since you have to compare chat/thinking/pro against an array of Anthropic and Google models.

But they publish all the same numbers, so you can make the full comparison yourself, if you want to.

Re:GPT-5.2 by Workaccount2 on Thursday December 11, 2025 @03:38PM (#46236816)
They are taking a page out of Apple's book.

Apple only compares to themselves. They don't even acknowledge the existence of others.

Re:GPT-5.2 by poormathskills on Thursday December 11, 2025 @01:30PM (#46235094)
OpenAI has never compared their models to models from other labs in their blog post. Open literally any past model launch post to see that.
Re:GPT-5.2 by boole1854 on Thursday December 11, 2025 @02:11PM (#46235745)
https://openai.com/index/hello-gpt-4o/

I see evaluations compared with Claude, Gemini, and Llama there on the GPT 4o post.

Re:GPT-5.2 by kgwgk on Thursday December 11, 2025 @02:30PM (#46236028)
“You are absolutely right, and I apologize for the confusion.”