News for hackers. Stuff that matters.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by lossolo on Friday December 20, 2024 @03:31PM (#42474745)
> making the most interesting and challenging LLM benchmark so far.

This[1] is currently the most challenging benchmark. I would like to see how O3 handles it, as O1 solved only 1%.

1. https://epoch.ai/frontiermath/the-benchmark

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by modeless on Friday December 20, 2024 @04:10PM (#42475068)
You're right, I was wrong to say "most challenging". I think the correct statement would be "most challenging long-standing benchmark" as I don't believe any other test designed in 2019 has resisted progress for so long.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by esafak on Saturday December 21, 2024 @09:07AM (#42479758)
They should put some famous, unsolved problems in the next edition so ML researchers do some actually useful work while they're "gaming" the benchmarks :)
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by modeless on Monday December 23, 2024 @04:41PM (#42497833)
I'm certain that the big labs will be gunning for the Millenium Prize problems.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by pynappo on Friday December 20, 2024 @03:44PM (#42474838)
Apparently o3 scored about 25%

https://youtu.be/SKBG1sqdyIU?t=4m40s

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by FiberBundle on Friday December 20, 2024 @04:06PM (#42475025)
This is actually the result that I find way more impressive. Elite mathematicians think these problems are challenging and thought they were years away from being solvable by AI.