Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB
by dot1x on Saturday December 21, 2024 @04:25AM
(#42478503)
It's all scam. ClosedAI trained on the data they were tested on, so no, nothing here is impressive.
[ Parent
| on HN ]
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB
by Tanjreeve on Sunday December 22, 2024 @04:48AM
(#42485379)
They definitely did or they probably did? Is there any source for that just so I can point It out to people?
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB
by catalypso on Sunday December 22, 2024 @10:07AM
(#42486771)
Just a clarification, they tuned on the public training dataset, not the semi-private one. The 87.5% score was on the semi-private eval, which means the model was still able to generalize well.
That being said, the fact that this is not a "raw" base model, but one tuned on the ARC-AGI tests distribution takes away from the impressiveness of the result — How much ? — I'm not sure, we'd need the un-tuned base o3 model score for that.
In the meantime, comparing this tuned o3 model to other un-tuned base models is unfair (apples-to-oranges kind of comparison).