News for hackers. Stuff that matters.

A comment on OpenAI O3 breakthrough high score on ARC-AGI-PUB. Parent comment

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by csomar on Friday December 20, 2024 @09:20PM (#42477020)
I don't care about benchmarks. O1 ranks higher than Claude on "benchmarks" but performs worse on particular real life coding situations. I'll judge the model myself by how useful/correct it is for my tasks rather than a hypothetical benchmarks.
[ Parent | on HN ]
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by og_kalu on Friday December 20, 2024 @11:18PM (#42477517)
In most non-competitive coding benchmarks (aider, live bench, swe-bench), o1 ranks worse than Sonnet (so the benchmarks aren't saying anything different) or at least did, the new checkpoint 2 days ago finally pushed o1 over sonnet on livebench.
[ Parent | on HN ]
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by tigershark on Friday December 20, 2024 @11:25PM (#42477545)
As I said, o3 demonstrated field medal level research capacity in the frontier math tests. But I’m sure that your use cases are much more difficult than that, obviously.
[ Parent | on HN ]
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by riku_iki on Saturday December 21, 2024 @11:53AM (#42480696)
there are many comments in internet about this, that only subset of frontier math benchmark is "field medal level research", and o3 likely scored on easier subset.

Also, all that stuff is shady in the way that it is just numbers from OAI, which are not reproducible on benchmark sponsored by OAI. If we say OAI could be bad actor, they had plenty of opportunities to cheat on this.

[ Parent | on HN ]
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by whynotminot on Friday December 20, 2024 @09:26PM (#42477046)
“Objective benchmarks are useless, let’s argue about which one works better for me personally.”
[ Parent | on HN ]
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by csomar on Friday December 20, 2024 @10:55PM (#42477440)
Yes. My benchmarks and their benchmarks means AGI. Their benchmarks only means over-fitted.
[ Parent | on HN ]
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by whynotminot on Friday December 20, 2024 @11:02PM (#42477466)
Ok so what if we get different results for our own personal benchmarks/use cases.

(See why objective benchmarks exist?)

[ Parent | on HN ]
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by bakugo on Friday December 20, 2024 @09:54PM (#42477176)
Yes, "objective" benchmarks can be gamed, real-life tasks cannot.
[ Parent | on HN ]