Re:Getting 50% (SoTA) on Arc-AGI with GPT-4o
by gwern on Monday June 17, 2024 @08:42PM
(#40712888)
From Chollet's perspective, the LLM hype started well before, with at least GPT-2 half a year before his paper, and he spent plenty of time mocking GPT-2 on Twitter before he came up with ARC as a rebuttal.
[ Parent
| on HN ]
Re:Getting 50% (SoTA) on Arc-AGI with GPT-4o
by modeless on Monday June 17, 2024 @11:04PM
(#40713697)
It's a very convincing rebuttal considering that GPT-3 and GPT-4 came out after ARC but made no significant progress on it. He seemingly had the single most accurate and verifiable prediction of anyone in the world (in 2019) about exactly what type of tasks scaled LLMs would be bad at.
Re:Getting 50% (SoTA) on Arc-AGI with GPT-4o
by gwern on Wednesday June 19, 2024 @11:52AM
(#40729440)
Well, that's true inasmuch as every other prediction did far worse. Saying ARC did the best is passing a low bar when your competition is people like Gary Marcus or HNers saying 'yeah but no scaled-up GPT-3 could ever write a whole program'...
But since ARC was from the start clearly a vision task - most of these transforms or rules make no sense without a visual geometric prior - it wasn't that convincing, and we see plenty of progress with LLMs.