News for hackers. Stuff that matters.
Re:GPT-5.6 by monk_grilla on Thursday July 09, 2026 @07:07PM (#48853590)
This is the first I have herd of this benchmark. Can someone explain how it in any way indicates how close we are to "AGI"?

Replay of Sol attempting the game: https://arcprize.org/replay/83543d22-8e1e-439a-8809-129ff1d9...

It seems a weird and arbitrary challenge for a language model to be expected to perform. It also seems like there are some harness/visual issues even in the first few steps, where it states that it hasn't moved when it clearly has.

Re:GPT-5.6 by drdrey on Thursday July 09, 2026 @10:13PM (#48854959)
the problems are general and abstract, domain specific knowledge and memorization don't help. Figuring out the rules, the goal, the controls, and how to solve in a reasonable budget all indicate some level of general ability.
Re:GPT-5.6 by andai on Friday July 10, 2026 @02:23AM (#48856423)
So basically they're well suited for like, an octopus or a crow?

I was thinking about those species earlier in the context of, what does intelligence mean outside of language.

The benchmark appears to be testing the same thing. Although I don't know how much transfer there would be between this data set and the kind of situations a crow or an octopus would encounter.

Edit: Huh, it's just a Game boy game? I just did a couple of the tasks. It looks like C64 era game to me. Navigating levels. A lot of overlap with animal intelligence then.

Re:GPT-5.6 by andriy_koval on Friday July 10, 2026 @01:14PM (#48862632)
> Can someone explain how it in any way indicates how close we are to "AGI"?

I think it is historical name. At some point when benchmarking was very undeveloped, this was targeting abstract reasoning and generalization, hence AGI.