Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB
by adamgordonbell on Friday December 20, 2024 @08:23PM
(#42476766)
So I'm thinking of something like Locked-room mystery where the idea is it's solvable, and the reader is given a chance to solve.
The reason it seems like an interesting bench, is it's a puzzle presented in a long context. Its like testing if an LLm is at Sherlock Holmes level of world and motivation modelling.
I would think this is a not so good bench. Author does not write logically, they write for entertainment.