Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB
by throwaway71271 on Friday December 20, 2024 @01:30PM
(#42473486)
| Name | Semi-private eval | Public eval |
|--------------------------------------|-------------------|-------------|
| Jeremy Berman | 53.6% | 58.5% |
| Akyürek et al. | 47.5% | 62.8% |
| Ryan Greenblatt | 43% | 42% |
| OpenAI o1-preview (pass@1) | 18% | 21% |
| Anthropic Claude 3.5 Sonnet (pass@1) | 14% | 21% |
| OpenAI GPT-4o (pass@1) | 5% | 9% |
| Google Gemini 1.5 (pass@1) | 4.5% | 8% |
https://arxiv.org/pdf/2412.04604
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB
by kandesbunzler on Friday December 20, 2024 @02:43PM
(#42474264)
why is this missing the o1 release / o1 pro models? Would love to know how much better they are
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB
by Freebytes on Friday December 20, 2024 @09:15PM
(#42476996)
This might be because they are referencing single step, and I do not think o1 is single step.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB
by aimanbenbaha on Friday December 20, 2024 @06:41PM
(#42476252)
Akyürek et al uses test-time compute.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB
by YetAnotherNick on Friday December 20, 2024 @01:31PM
(#42473494)
Here are the results for base models[1]:
o3 (coming soon) 75.7% 82.8%
o1-preview 18% 21%
Claude 3.5 Sonnet 14% 21%
GPT-4o 5% 9%
Gemini 1.5 4.5% 8%
Score (semi-private eval) / Score (public eval)
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB
by simonw on Friday December 20, 2024 @03:46PM
(#42474843)
I'd love to know how Claude 3.5 Sonnet does so well despite (presumably) not having the same tricks as the o-series models.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB
by Bjorkbat on Friday December 20, 2024 @04:39PM
(#42475326)
It's easy to miss, but if you look closely at the first sentence of the announcement they mention that they used a version of o3 trained on a public dataset of ARC-AGI, so technically it doesn't belong on this list.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB
by dot1x on Saturday December 21, 2024 @04:25AM
(#42478503)
It's all scam. ClosedAI trained on the data they were tested on, so no, nothing here is impressive.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB
by Tanjreeve on Sunday December 22, 2024 @04:48AM
(#42485379)
They definitely did or they probably did? Is there any source for that just so I can point It out to people?
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB
by catalypso on Sunday December 22, 2024 @10:07AM
(#42486771)
Just a clarification, they tuned on the public training dataset, not the semi-private one. The 87.5% score was on the semi-private eval, which means the model was still able to generalize well.
That being said, the fact that this is not a "raw" base model, but one tuned on the ARC-AGI tests distribution takes away from the impressiveness of the result — How much ? — I'm not sure, we'd need the un-tuned base o3 model score for that.
In the meantime, comparing this tuned o3 model to other un-tuned base models is unfair (apples-to-oranges kind of comparison).
I wonder how well the latest Claude 3.5 Sonnet does on this benchmark and if it's near o1.