the problems are general and abstract, domain specific knowledge and memorization don't help. Figuring out the rules, the goal, the controls, and how to solve in a reasonable budget all indicate some level of general ability.
So basically they're well suited for like, an octopus or a crow?
I was thinking about those species earlier in the context of, what does intelligence mean outside of language.
The benchmark appears to be testing the same thing. Although I don't know how much transfer there would be between this data set and the kind of situations a crow or an octopus would encounter.
Edit: Huh, it's just a Game boy game? I just did a couple of the tasks. It looks like C64 era game to me. Navigating levels. A lot of overlap with animal intelligence then.
Re:GPT-5.6
by andriy_koval on Friday July 10, 2026 @01:14PM
(#48862632)
> Can someone explain how it in any way indicates how close we are to "AGI"?
I think it is historical name. At some point when benchmarking was very undeveloped, this was targeting abstract reasoning and generalization, hence AGI.
Replay of Sol attempting the game: https://arcprize.org/replay/83543d22-8e1e-439a-8809-129ff1d9...
It seems a weird and arbitrary challenge for a language model to be expected to perform. It also seems like there are some harness/visual issues even in the first few steps, where it states that it hasn't moved when it clearly has.