News for hackers. Stuff that matters.
Re:Arc Prize 2024 Winners and Technical Report by celeritascelery on Friday December 06, 2024 @03:24PM (#42343975)
What surprises me about this is how poorly general-purpose LLMs do. The best one is OpenAI o1-preview at 18%. This is significantly worse than the purpose-built models like ARChitects (which scored 53.5). This model used TTT to train on the ARC-AGI task specification (amoung other things). It seems that even if someone creates a model that can "solve" ARC, it still is not indicative of AGI since it is not "general" anymore, it is just specialized to this particular task. Similar to how chess engines are not AGI, despite being superhuman at chess. It will be much more convincing when general models not trained specifically for ARC can still score well on it.

They do mention that some of the tasks here are susceptible to brute force and they plan to address that in ARC-AGI-2.

> nearly half (49%) of the private evaluation set was solved by at least one team during the original 2020 Kaggle competition all of which were using some variant of brute-force program search. This suggests a large fraction of ARC-AGI-1 tasks are susceptible to this kind of method and does not carry much useful signal towards general intelligence.

Re:Arc Prize 2024 Winners and Technical Report by fchollet on Friday December 06, 2024 @03:40PM (#42344172)
It is correct that the first model that will beat ARC-AGI will only be able to handle ARC-AGI tasks. However, the idea is that the architecture of that model should be able to be repurposed to arbitrary problems. That is what makes ARC-AGI a good compass towards AGI (unlike chess).

For instance, current top models use TTT, which is a completely general-purpose technique that provides the most significant boost to DL model's generalization power in recent memory.

The other category of approach that is working well is program synthesis -- if pushed to the extent that it could solve ARC-AGI, the same system could be redeployed to solve arbitrary programming tasks, as well as tasks isomorphic to programming (such as theorem proving).

Re:Arc Prize 2024 Winners and Technical Report by scoobertdoobert on Friday December 06, 2024 @04:00PM (#42344382)
François, have you coded and tested a solution yourself that you think will work best?
Re:Arc Prize 2024 Winners and Technical Report by optimalsolver on Friday December 06, 2024 @08:10PM (#42346334)
Hey, he's the visionary. You come up with the nuts and bolts.
Re:Arc Prize 2024 Winners and Technical Report by homarp on Saturday December 07, 2024 @01:55AM (#42347795)
is keras nuts and bolts enough?
Re:Arc Prize 2024 Winners and Technical Report by ipunchghosts on Saturday December 07, 2024 @06:43PM (#42353768)
Keres is a good abstraction model but poorly implemented.
Re:Arc Prize 2024 Winners and Technical Report by ipunchghosts on Saturday December 07, 2024 @06:45PM (#42353779)
"However, the idea is that the architecture of that model should be able to be repurposed to arbitrary problems"

From a mathematical perspective, this doesn't sound right. All NNs are universal apprxomators and in theory can all learn the same thing to equal ability. It's more about the learning algorithm than the architecture IMO.

Re:Arc Prize 2024 Winners and Technical Report by mrandish on Friday December 06, 2024 @04:32PM (#42344692)
> It seems that even if someone creates a model that can "solve" ARC, it still is not indicative of AGI since it is not "general" anymore

I recently explained why I like ARC to a non-technical friend this way: "When an AI solves ARC it won't be proof of AGI. It's the opposite. As long as ARC remains unsolved I'm confident we're not even close to AGI."

For the sake of being provocative, I'd even argue that ARC remaining unsolved is a sign we're not yet making meaningful progress in the right direction. AGI is the top of Everest. ARC is base camp.

Re:Arc Prize 2024 Winners and Technical Report by iwsk on Friday December 06, 2024 @06:04PM (#42345543)
in other words, solving ARC is necessary but not sufficient for AGI
Re:Arc Prize 2024 Winners and Technical Report by mrandish on Friday December 06, 2024 @07:55PM (#42346258)
Yes! That's the exact phrase I would have used with someone on HN. But that doesn't describe my non-technical friend. :-)
Re:Arc Prize 2024 Winners and Technical Report by YeGoblynQueenne on Saturday December 07, 2024 @07:18AM (#42349286)
Why is it necessary? Could a spider solve ARC-AGI, or could a pigeon, or a cat? And if an animal doesn't need to solve ARC-AGI to be intelligent, then why does an AGI?
Re:Arc Prize 2024 Winners and Technical Report by thomasahle on Friday December 06, 2024 @07:58PM (#42346267)
> What surprises me about this is how poorly general-purpose LLMs do. The best one is OpenAI o1-preview at 18%.

o1-preview doesn't even have image input, so I wonder how they used it.

Also, Ryan Greenblatts solution basically does "best of 4000" iirc. Presumably o1-preview was single shot.

Re:Arc Prize 2024 Winners and Technical Report by celeritascelery on Friday December 06, 2024 @11:45PM (#42347333)
None of the models use images, they all operate and a json format the describes the input squares.