News for hackers. Stuff that matters.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by modeless on Friday December 20, 2024 @01:22PM (#42473419)
Congratulations to Francois Chollet on making the most interesting and challenging LLM benchmark so far.

A lot of people have criticized ARC as not being relevant or indicative of true reasoning, but I think it was exactly the right thing. The fact that scaled reasoning models are finally showing progress on ARC proves that what it measures really is relevant and important for reasoning.

It's obvious to everyone that these models can't perform as well as humans on everyday tasks despite blowout scores on the hardest tests we give to humans. Yet nobody could quantify exactly the ways the models were deficient. ARC is the best effort in that direction so far.

We don't need more "hard" benchmarks. What we need right now are "easy" benchmarks that these models nevertheless fail. I hope Francois has something good cooked up for ARC 2!

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by Balgair on Friday December 20, 2024 @02:38PM (#42474214)
Complete aside here: I used to do work with amputees and prosthetics. There is a standardized test (and I just cannot remember the name) that fits in a briefcase. It's used for measuring the level of damage to the upper limbs and for prosthetic grading.

Basically, it's got the dumbest and simplest things in it. Stuff like a lock and key, a glass of water and jug, common units of currency, a zipper, etc. It tests if you can do any of those common human tasks. Like pouring a glass of water, picking up coins from a flat surface (I chew off my nails so even an able person like me fails that), zip up a jacket, lock your own door, put on lipstick, etc.

We had hand prosthetics that could play Mozart at 5x speed on a baby grand, but could not pick up a silver dollar or zip a jacket even a little bit. To the patients, the hands were therefore about as useful as a metal hook (a common solution with amputees today, not just pirates!).

Again, a total aside here, but your comment just reminded me of that brown briefcase. Life, it turns out, is a lot more complex than we give it credit for. Even pouring the OJ can be, in rare cases, transcendent.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by ubj on Friday December 20, 2024 @02:54PM (#42474389)
There's a lot of truth in this. I sometimes joke that robot benchmarks should focus on common household chores. Given a basket of mixed laundry, sort and fold everything into organized piles. Load a dishwasher given a sink and counters overflowing with dishes piled up haphazardly. Clean a bedroom that kids have trashed. We do these tasks almost without thinking, but the unstructured nature presents challenges for robots.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by zamalek on Friday December 20, 2024 @04:00PM (#42474969)
Slightly tangential, we already have amazing laundry robots. They are called washing and drying machines. We don't give these marvels enough credit, mostly because they aren't shaped like humans.

Humanoid robots are mostly a waste of time. Task-shaped robots are much easier to design, build, and maintain... and are more reliable. Some of the things you mention might needs humanoid versatility (loading the dishwasher), others would be far better served by purpose-built robots (laundry sorting).

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by jkaptur on Friday December 20, 2024 @04:04PM (#42475009)
I'm embarrassed to say that I spent a few moments daydreaming about a robot that could wash my dishes. Then I thought about what to call it...
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by musicale on Friday December 20, 2024 @04:23PM (#42475184)
Sadly current "dishwasher" models are neither self-loading nor unloading. (Seems like they should be able to take a tray of dishes, sort them, load them, and stack them after cleaning.)

Maybe "busbot" or "scullerybot".

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by vidarh on Friday December 20, 2024 @05:12PM (#42475582)
The problem is more doing it in sufficiently little space, and using little enough water and energy. Doing one that you just feed dishes individually and that immediate wash them and feed them to storage should be entirely viable, but it'd be wasteful, and it'd compete with people having multiple small drawer-style dishwashers, offering relatively little convenience over that.

It seems most people aren't willing to pay for multiple dishwashers - even multiple small ones or set aside enough space, and that places severe constraints on trying to do better.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by wsintra2022 on Friday December 20, 2024 @04:32PM (#42475255)
Was it a dishwasher? Just give it all your unclean dishes and tell it to go, come back an hour later and they all washed and mostly dried!
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by rytis on Friday December 20, 2024 @04:27PM (#42475216)
I agree. I don’t know where this obsession comes from. Obsession with resembling as close to humans as possible. We’re so far from being perfect. If you need proof just look at your teeth. Yes, we’re relatively universal, but a screwdriver is more efficient at driving in screws that our fingers. So please, stop wasting time building perfect universal robots, build more purpose-build ones.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by golol on Friday December 20, 2024 @05:05PM (#42475535)
The shape doesn't matter! Non-humanoid shapes give minir advantages on specific tasks but for a general robot you'll have a hard time finding a shape much more optimal than humanoid. And if you go with humanoid you have so much data available! Videos contain the information of which movements a robot should execude. Teleoperation is easy. This is the bitter lesson! The shape doesn't matter, any shape will work with the right architecture, data and training!
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by Nevermark on Friday December 20, 2024 @04:40PM (#42475328)
Given we have shaped so many tasks to fit our bodies, it will be a long time before a bot able to do a variety/majority of human tasks the human way won’t be valuable.

1000 machines specialized for 1000 tasks are great, but don’t deliver the same value as a single bot that can interchange with people flexibly.

Costly today, but wont be forever.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by rowanG077 on Friday December 20, 2024 @05:44PM (#42475852)
Purpose build robots are basically solved. Dishwashers, laundry machines, assembly robots, etc. the moat is a general purpose robot that can do what a human can do.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by graemep on Friday December 20, 2024 @05:07PM (#42475551)
Great examples. They are simple, reliable, efficient and effective. Far better than blindly copying what a human being does. Maybe there are equally clever ways of doing things like folding clothes.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by Geee on Saturday December 21, 2024 @11:45AM (#42480628)
There isn't a "task-shaped" robot for unstructured and complex manipulation, other than high DoF arms with vision and neural nets. For example, a machine which can cook food would be best solved with two robotic arms. However, these stationary arms would be wasted if they were just idling most of the time. So, you add locomotion and dynamic balancing with legs. And now these two arms can be used in 1000 different tasks, which makes them 1000x more valuable.

So, not only is the human form the only solution for many tasks, it's also a much cheaper solution considering the idle time of task-specific robots. You would need only a single humanoid robot for all tasks, instead of buying a different machine for each task. And instead of having to design and build a new machine for each task, you'll need to just download new software for each task.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by Balgair on Friday December 20, 2024 @03:40PM (#42474804)
I maintain that whoever invents a robust laundry folding robot will be a trillionaire. In that, I dump jumbled clean clothes straight from a dryer at it and out comes folded and sorted clothes (and those loner socks). I know we're getting close, but I also know we're not there yet.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by smokel on Friday December 20, 2024 @03:56PM (#42474932)
We are certainly getting close! In 2010, watching PR2 fold some unseen towels is similar to watching paint dry [1], but we can now enjoy robots attain lazy student-level laundry folding in real-time, as demonstrated by π₀[2].

[1] https://www.youtube.com/watch?v=gy5g33S0Gzo

[2] https://www.physicalintelligence.company/blog/pi0

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by yongjik on Friday December 20, 2024 @03:48PM (#42474862)
I can live without folding laundry (I can just shove my undershirts in the closet, who cares if it's not folded), but whoever manufactures a reliable auto-loading dishwasher will have my dollars. Like, just put all your dishes in the sink and let the machine handle them.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by Brybry on Friday December 20, 2024 @04:34PM (#42475282)
But if your dishwasher is empty is takes nearly the same amount of time/effort to put dishes straight into the dishwasher that it does to put them in the sink.

I think I'd only really save time by having a robot that could unload my dishwasher and put up the clean dishes.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by namibj on Friday December 20, 2024 @04:47PM (#42475390)
That's called a second dishwasher: one is for taking out, the other for putting in. When the latter is full, turn it on, dirty dishes wait outside until the cycle finishes, when the dishwashers switch roles.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by ptsneves on Friday December 20, 2024 @05:15PM (#42475611)
I thought about this and it gets even better. You do not really need shelves as you just use the clean dishwasher as the storage place. I honestly don’t know why this is not a thing in big or wealthy homes.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by jannyfer on Friday December 20, 2024 @05:35PM (#42475780)
Another thing that bothers me is that dishwashers are low. As I get older, I’m finding it really annoying to bend down.

So get me a counter-level dishwasher cabinet and I’ll be happy!

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by oangemangut on Friday December 20, 2024 @06:07PM (#42476025)
We have a double drawer dishwasher and it hurts my brain watching friends plan around their nightly wash.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by yongjik on Friday December 20, 2024 @05:17PM (#42475623)
Hmm, that doesn't match my experience. It takes me a lot more time to put dishes into the dishwasher, because it has different places for cutlery, bowls, dishes, and so on, and of course the existing structure never matches my bowls' size perfectly so I have to play tetris or run it with only 2/3 filled (which will cause me to waste more time as I have to do dishes again sooner).

And that's before we get to bits of sticky rice left on bowls, which somehow dishwashers never scrape off clean. YMMV.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by HPsquared on Friday December 20, 2024 @05:59PM (#42475950)
1. Get a set of dishes that does fit nicely together in the dishwasher.

2. Start with a cold prewash, preferably with a little powder in there too. This massively helps with stubborn stuff. This one is annoying though because you might have to come back and switch it on after the prewash. A good job for the robot butler.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by xnx on Saturday December 21, 2024 @04:15PM (#42482366)
Buy one bowl you like (I use a silicone one) and use it for everything. Rarely requires more than a quick rinse.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by sssilver on Sunday December 22, 2024 @12:49AM (#42484634)
Why can't dishwashers just be small, single-dish appliances in which you put the plate/mug/wine glass/forks/whatever, close it, push a button, and 10 seconds later it's clean and dry, you unload and repeat?
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by dweekly on Friday December 20, 2024 @04:06PM (#42475031)
I was a believer in Gal's FoldiMate but sadly it...folded.

https://en.m.wikipedia.org/wiki/FoldiMate

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by sss111 on Friday December 20, 2024 @04:01PM (#42474976)
Honestly, a robot that can hang jumbled clean clothes instead of folding them would be good enough, it's crazy how we don't even have those.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by jessekv on Friday December 20, 2024 @03:47PM (#42474851)
I want it to lay out an outfit every day too. Hopefully without hallucination.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by stefs on Friday December 20, 2024 @03:56PM (#42474931)
it's not hallucination, it's high fashion
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by tanseydavid on Friday December 20, 2024 @05:09PM (#42475561)
Yes, but the stupid robot laid out your Thursday-black-Turtleneck for you on Saturday morning. That just won't suffice.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by oblio on Friday December 20, 2024 @03:41PM (#42474812)
Laundry folding and laundry ironing, I would say.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by musicale on Friday December 20, 2024 @04:40PM (#42475335)
Hopefully will detect whether a small child is inside or not.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by nradov on Friday December 20, 2024 @03:49PM (#42474871)
There is the Foldimate robot. I don't know how well it works. It doesn't seem to pair up socks. (Deleted the web link, it might not be legitimate.)
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by smokel on Friday December 20, 2024 @04:05PM (#42475018)
Beware, this website is probably a scam.

Foldimate has gone bankrupt in 2021 [1], and the domain referral from foldimate.com to a 404 page at miele.com, suggests that it was Miele who bought up the remains, not a sketchy company with a ".website" top-level domain.

[1] https://en.wikipedia.org/wiki/FoldiMate

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by imafish on Friday December 20, 2024 @03:47PM (#42474850)
> I maintain that whoever invents a robust laundry folding robot will be a trillionaire

… so Elon Musk? :D

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by blargey on Friday December 20, 2024 @04:35PM (#42475291)
At this point I'm not sure we'll actually get a task-specific machine for laundry folding/sorting before humanoid robots gain the capability to do it well enough.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by drdrey on Friday December 20, 2024 @04:13PM (#42475097)
I think assembling Legos would be a cool robot benchmark: you need to parse the instructions, locate the pieces you need, pick them up, orient them, snap them to your current assembly, visually check if you achieved the desired state, repeat
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by serpix on Saturday December 21, 2024 @01:39AM (#42477919)
I agree. Watching my toddler daughter build with small legos makes me understand how incredible fine motor skills are as even with small fingers some of the blocks are just too hard to snap together.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by ecshafer on Friday December 20, 2024 @02:56PM (#42474401)
I had a pretty bad case of tendinitis once, that basically made my thumb useless since using it would cause extreme pain. That test seems really good. I could use a computer keyboard without any issue, but putting a belt on or pouring water was impossible.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by vidarh on Friday December 20, 2024 @05:13PM (#42475591)
I had a swollen elbow a short while ago, and the amount of things I've never thought about that were affected by reduced elbow join mobility and an inability to put pressure on the elbow was disturbing.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by alexose on Friday December 20, 2024 @03:24PM (#42474657)
It feels like there's a whole class of information that easily shorthanded, but really hard to explain to novices.

I think a lot about carpentry. From the outside, it's pretty easy: Just make the wood into the right shape and stick it together. But as one progresses, the intricacies become more apparent. Variations in the wood, the direction of the grain, the seasonal variations in thickness, joinery techniques that are durable but also time efficient.

The way this information connects is highly multisensory and multimodal. I now know which species of wood to use for which applications. This knowledge was hard won through many, many mistakes and trials that took place at my home, the hardware store, the lumberyard, on YouTube, from my neighbor Steve, and in books written by experts.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by croemer on Friday December 20, 2024 @03:06PM (#42474497)
> We had hand prosthetics that could play Mozart at 5x speed on a baby grand, but could not pick up a silver dollar or zip a jacket even a little bit. "

I must be missing something, how can they be able to play Mozart at 5x speed with their prosthetics but not zip a jacket? They could press keys but not do tasks requiring feedback?

Or did you mean they used to play Mozart at 5x speed before they became amputees?

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by rahimnathwani on Friday December 20, 2024 @03:25PM (#42474667)
Imagine a prosthetic 'hand' that has 5 regular fingers, rather than 4 fingers and a thumb. It would be able to play a piano just fine, but be unable to grasp anything small, like a zipper.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by ben_w on Friday December 20, 2024 @03:29PM (#42474725)
Playing a piano involves pushing down on the right keys with the right force at the right time, but that could be pre-programmed well before computers. The self-playing piano in the saloon in Westworld wasn't a huge anachronism, such things slightly overlapped with the Wild West era: https://en.wikipedia.org/wiki/Player_piano

Picking up a 1mm thick metal disk from a flat surface requires the user gives the exact time, place, and force, and I'm not even sure what considerations it needs for surface materials (e.g. slightly squishy fake skin) and/or tip shapes (e.g. fake nails).

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by numpad0 on Friday December 20, 2024 @03:35PM (#42474777)
> Picking up a 1mm thick metal disk from a flat surface requires the user gives the exact time, place, and force

place sure but can't you cheat a bit for time and force with compliance("impedance control")?

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by ben_w on Friday December 20, 2024 @04:22PM (#42475181)
In theory, apparently not in practice.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by 8note on Friday December 20, 2024 @03:28PM (#42474711)
zipping up a jacket is really hard to do, and requires very precise movements and coordination between hands.

playing mozart is much more forgiving in terms of the number of different motions you have to make in different directions, the amount of pressure to apply, and even the black keys are much bigger than large sized zipper tongues.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by Balgair on Friday December 20, 2024 @03:38PM (#42474793)
Pretty much. The issue with zippers is that the fabric moves about in unpredictable ways. Piano playing was just movement programs. Zipping required (surprisingly) fast feedback. Also, gripping is somewhat tough compared to pressing.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by numpad0 on Friday December 20, 2024 @03:27PM (#42474696)
Thumb not opposable?
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by oblio on Friday December 20, 2024 @03:40PM (#42474810)
I'm far from a piano player, but I can definitely push piano buttons quite quickly while zipping up my jacket when it's cold and/or wet outside is really difficult.

Even more so for picking up coins from a flat surface.

For robotics, it's kind of obvious, speed is rarely an issue, so the "5x" part is almost trivial. And you can program the sequence quite easily, so that's also doable. Piano keys are big and obvious and an ergonomically designed interface meant to be relatively easy to press, ergo easy even for a prosthetic. A small coin on a flat surface is far from ergonomic.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by yongjik on Friday December 20, 2024 @03:53PM (#42474913)
I play piano as a hobby, and the funny thing is, if my hands are so cold that I can't zip up my jacket, there's no way I can play anything well. I know it's not quite zipping up jackets ;) but a human playing the piano does require a fast feedback loop.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by croemer on Friday December 20, 2024 @03:46PM (#42474846)
But how do you deliberately control those fingers to actually play yourself what you have in mind rather than something preprogrammed? Surely the idea of a prosthetic does not just mean "a robot that is connected to your body", but something that the owner control with your mind.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by vidarh on Friday December 20, 2024 @05:18PM (#42475635)
Nobody said anything about deliberately controlling those fingers to play yourself. Clearly it's not something you do for the sake of the enjoyment of playing, but more likely a demonstration of the dexterity of the prosthesis and ability to program it for complex tasks.

The idea of a prosthesis is to help you regain functionality. If the best way of doing that is through automation, then it'd make little sense not to.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by n144q on Friday December 20, 2024 @05:42PM (#42475842)
Well, you see, while the original comment says they could play at 5x speed, it does not say it plays at that speed well or play it beautifully. Any teacher or any student who learned piano for a while will tell you that this matters a lot, especially for classical music -- being able to accurately play at an even tempo with the correct dynamics and articulation is hard and is what differentiates a beginner/intermediate player from an advanced one. In fact, one mistake many students make is playing a piece too fast when they are not ready, and teachers really want students to practice very slowly.

My point is -- being able to zip a jacket is all about those subtle actions, and could actually be harder than "just" playing piano fast.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by m463 on Friday December 20, 2024 @02:54PM (#42474384)
It would be interesting to see trick questions.

Like in your test

a hand grenade and a pin - don't pull the pin.

Or maybe a mousetrap? but maybe that would be defused?

in the ai test...

or Global Thermonuclear War, the only winning move is...

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by HPsquared on Friday December 20, 2024 @03:12PM (#42474556)
Gaming streams being in the training data, it might pull the pin because "that's what you do".
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by 8note on Friday December 20, 2024 @03:29PM (#42474723)
or, because it has to give an output, and pulling the pin is the only option
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by TeMPOraL on Friday December 20, 2024 @05:24PM (#42475692)
There's also the option of not pulling the pin, and shooting your enemies as they instinctively run from what they think is a live grenade. Saw it on a TV show the other day.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by sdenton4 on Friday December 20, 2024 @03:12PM (#42474553)
to move first!
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by m463 on Friday December 20, 2024 @05:44PM (#42475851)
oh crap. lol!
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by Method-X on Friday December 20, 2024 @03:27PM (#42474695)
Was it the Southampton hand assessment procedure?
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by Balgair on Friday December 20, 2024 @03:35PM (#42474774)
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by oblio on Friday December 20, 2024 @03:37PM (#42474790)
This was actually discovered quite early on in the history of AI:

> Rodney Brooks explains that, according to early AI research, intelligence was "best characterized as the things that highly educated male scientists found challenging", such as chess, symbolic integration, proving mathematical theorems and solving complicated word algebra problems. "The things that children of four or five years could do effortlessly, such as visually distinguishing between a coffee cup and a chair, or walking around on two legs, or finding their way from their bedroom to the living room were not thought of as activities requiring intelligence."

https://en.wikipedia.org/wiki/Moravec%27s_paradox

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by bawolff on Friday December 20, 2024 @03:55PM (#42474924)
I don't know why people always feel the need to gender these things. Highly educated female scientists generally find the same things challenging.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by robocat on Friday December 20, 2024 @04:26PM (#42475209)
I don't know why anyone would blame people as though someone is making an explicit choice. I find your choice of words to be insulting to the OP.

We learn our language and stereotypes subconciously from our society, and it is no easy thing to fight against that.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by Barrin92 on Friday December 20, 2024 @05:12PM (#42475576)
>I don't know why people always feel the need to gender these things

Because it's relevant to the point being made, i.e. that these tests reflect the biases and interests of the people who make them. This is true not just for AI tests, but intelligence test applied to humans. That Demis Hassabis, a chess player and video game designer, decided to test his machine on video games, Go and chess probably is not an accident.

The more interesting question is why people respond so apprehensively to pointing out a very obvious problem and bias in test design.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by bawolff on Friday December 20, 2024 @06:55PM (#42476335)
> i.e. that these tests reflect the biases and interests of the people who make them

Of course. However i believe we can't move past that without being honest about where these biases are coming from. Many things in our world are the result of gender bias, both subtle and overt. However, at least at first glance, this does not appear to be one of them, and statements like the grandparent's quote serve to perpetuate such biases further.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by oblio on Saturday December 21, 2024 @04:34AM (#42478556)
It's a quote from the 80s from the original author (who is a man...)...

Thank you for virtue signalling, though.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by bawolff on Saturday December 21, 2024 @04:48AM (#42478625)
> It's a quote from the 80s from the original author (who is a man...)...

Yes, that was pretty clear in the original comment (?)

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by oblio on Saturday December 21, 2024 @04:51AM (#42478639)
Then remove the parts that offend your modern sensibilities and focus on the essence.

He was right. Scientists were focusing on the "science-y" bits and completely missed the elephant in the room, that the thing a toddler already masters are the monster challenge for AI right now, before we even get into "meaning of life" type stuff.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by CooCooCaCha on Friday December 20, 2024 @03:00PM (#42474430)
That’s why the goal isn’t just benchmark scores, it’s reliable and robust intelligence.

In that sense, the goalposts haven’t moved in a long time despite claims from AI enthusiasts that people are constantly moving goalposts.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by throwup238 on Friday December 20, 2024 @04:53PM (#42475434)
This is expressed in AI research as Moravec's paradox: https://en.wikipedia.org/wiki/Moravec%27s_paradox

Getting to LLMs that could talk to us turned out to be a lot easier than making something that could control even a robotic arm without precise programming, let alone a humanoid.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by MarcelOlsz on Friday December 20, 2024 @05:14PM (#42475603)
>We had hand prosthetics that could play Mozart at 5x speed on a baby grand

I'd love to know more about this.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by xnx on Friday December 20, 2024 @05:37PM (#42475804)
Despite lake of fearsome teeth or claws, humans are way op due to brain, hand dexterity, and balance.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by dang on Friday December 20, 2024 @06:15PM (#42476076)
We detached this subthread from https://news.ycombinator.com/item?id=42473419

(nothing wrong with it! I'm just trying to prune the top subthread)

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by zone411 on Friday December 20, 2024 @04:17PM (#42475136)
It's the least interesting benchmark for language models among all they've released, especially now that we already had a large jump in its best scores this year. It might be more useful as a multimodal reasoning task since it clearly involves visual elements, but with o3 already performing so well, this has proven unnecessary. ARC-AGI served a very specific purpose well: showcasing tasks where humans easily outperformed language models, so these simple puzzles had their uses. But tasks like proving math theorems or programming are far more impactful.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by versteegen on Friday December 20, 2024 @07:12PM (#42476408)
ARC wasn't designed as a benchmark for LLMs, and it doesn't make much sense to compare them on it since it's the wrong modality. Even a MLM with image inputs can't be expected to do well, since they're nothing like 99.999% of the training data. The fact that even a text-only LLM can solve ARC problems with the proper framework is important, however.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by jug on Friday December 20, 2024 @03:59PM (#42474960)
I liked the SimpleQA benchmark that measures hallucinations. OpenAI models did surprisingly poorly, even o1. In fact, it looks like OpenAI often does well on benchmarks by taking the shortcut to be more risk prone than both Anthropic and Google.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by skywhopper on Friday December 20, 2024 @03:36PM (#42474781)
"The fact that scaled reasoning models are finally showing progress on ARC proves that what it measures really is relevant and important for reasoning."

Not sure I understand how this follows. The fact that a certain type of model does well on a certain benchmark means that the benchmark is relevant for a real-world reasoning? That doesn't make sense.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by munchler on Friday December 20, 2024 @04:27PM (#42475219)
It shows objectively that the models are getting better at some form of reasoning, which is at least worth noting. Whether that improved reasoning is relevant for the real world is a different question.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by moffkalast on Friday December 20, 2024 @05:00PM (#42475494)
It shows objectively that one model got better at this specific kind of weird puzzle that doesn't translate to anything because it is just a pointless pattern matching puzzle that can be trained for, just like anything else. In fact they specifically trained for it, they say so upfront.

It's like the modern equivalent of saying "oh when AI solves chess it'll be as smart as a person, so it's a good benchmark" and we all know how that nonsense went.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by munchler on Friday December 20, 2024 @05:31PM (#42475745)
Hmm, you could be right, but you could also be very wrong. Jury's still out, so the next few years will be interesting.

Regarding the value of "pointless pattern matching" in particular, I would refer you to Douglas Hofstadter's discussion of Bongard problems starting on page 652 of _Godel, Escher, Bach_. Money quote: "I believe that the skill of solving Bongard [pattern recognition] problems lies very close to the core of 'pure' intelligence, if there is such a thing."

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by moffkalast on Friday December 20, 2024 @07:02PM (#42476373)
Well I certainly at least agree with that second part, the doubt if there is such a thing ;)

The problem with pattern matching of sequences and transformers as an architecture is that it's something they're explicitly designed to be good at with self attention. Translation is mainly matching patterns to equivalents in different languages, and continuing a piece of text is following a pattern that exists inside it. This is primarily why it's so hard to draw a line between what an LLM actually understands and what it just wings naturally through pattern memorization and why everything about them is so controversial.

Honestly I was really surprised that all models did so poorly on ARC in general thus far, since it really should be something they ought to be superhuman at from the get-go. Probably more of a problem that it's visual in concept than anything else.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by bagels on Friday December 20, 2024 @07:21PM (#42476461)
It doesn't follow, faulty logic. The two are probably correlated though.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by lossolo on Friday December 20, 2024 @03:31PM (#42474745)
> making the most interesting and challenging LLM benchmark so far.

This[1] is currently the most challenging benchmark. I would like to see how O3 handles it, as O1 solved only 1%.

1. https://epoch.ai/frontiermath/the-benchmark

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by modeless on Friday December 20, 2024 @04:10PM (#42475068)
You're right, I was wrong to say "most challenging". I think the correct statement would be "most challenging long-standing benchmark" as I don't believe any other test designed in 2019 has resisted progress for so long.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by esafak on Saturday December 21, 2024 @09:07AM (#42479758)
They should put some famous, unsolved problems in the next edition so ML researchers do some actually useful work while they're "gaming" the benchmarks :)
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by modeless on Monday December 23, 2024 @04:41PM (#42497833)
I'm certain that the big labs will be gunning for the Millenium Prize problems.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by pynappo on Friday December 20, 2024 @03:44PM (#42474838)
Apparently o3 scored about 25%

https://youtu.be/SKBG1sqdyIU?t=4m40s

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by FiberBundle on Friday December 20, 2024 @04:06PM (#42475025)
This is actually the result that I find way more impressive. Elite mathematicians think these problems are challenging and thought they were years away from being solvable by AI.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by dtquad on Friday December 20, 2024 @01:26PM (#42473449)
Are there any single-step non-reasoner models that do well on this benchmark?

I wonder how well the latest Claude 3.5 Sonnet does on this benchmark and if it's near o1.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by throwaway71271 on Friday December 20, 2024 @01:30PM (#42473486)

    | Name                                 | Semi-private eval | Public eval |
    |--------------------------------------|-------------------|-------------|
    | Jeremy Berman                        | 53.6%             | 58.5%       |
    | Akyürek et al.                       | 47.5%             | 62.8%       |
    | Ryan Greenblatt                      | 43%               | 42%         |
    | OpenAI o1-preview (pass@1)           | 18%               | 21%         |
    | Anthropic Claude 3.5 Sonnet (pass@1) | 14%               | 21%         |
    | OpenAI GPT-4o (pass@1)               | 5%                | 9%          |
    | Google Gemini 1.5 (pass@1)           | 4.5%              | 8%          |

https://arxiv.org/pdf/2412.04604
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by kandesbunzler on Friday December 20, 2024 @02:43PM (#42474264)
why is this missing the o1 release / o1 pro models? Would love to know how much better they are
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by Freebytes on Friday December 20, 2024 @09:15PM (#42476996)
This might be because they are referencing single step, and I do not think o1 is single step.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by aimanbenbaha on Friday December 20, 2024 @06:41PM (#42476252)
Akyürek et al uses test-time compute.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by YetAnotherNick on Friday December 20, 2024 @01:31PM (#42473494)
Here are the results for base models[1]:

  o3 (coming soon)  75.7% 82.8%
  o1-preview        18%   21%
  Claude 3.5 Sonnet 14%   21%
  GPT-4o            5%    9%
  Gemini 1.5        4.5%  8%
Score (semi-private eval) / Score (public eval)

[1]: https://arcprize.org/2024-results

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by simonw on Friday December 20, 2024 @03:46PM (#42474843)
I'd love to know how Claude 3.5 Sonnet does so well despite (presumably) not having the same tricks as the o-series models.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by Bjorkbat on Friday December 20, 2024 @04:39PM (#42475326)
It's easy to miss, but if you look closely at the first sentence of the announcement they mention that they used a version of o3 trained on a public dataset of ARC-AGI, so technically it doesn't belong on this list.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by dot1x on Saturday December 21, 2024 @04:25AM (#42478503)
It's all scam. ClosedAI trained on the data they were tested on, so no, nothing here is impressive.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by Tanjreeve on Sunday December 22, 2024 @04:48AM (#42485379)
They definitely did or they probably did? Is there any source for that just so I can point It out to people?
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by catalypso on Sunday December 22, 2024 @10:07AM (#42486771)
Just a clarification, they tuned on the public training dataset, not the semi-private one. The 87.5% score was on the semi-private eval, which means the model was still able to generalize well.

That being said, the fact that this is not a "raw" base model, but one tuned on the ARC-AGI tests distribution takes away from the impressiveness of the result — How much ? — I'm not sure, we'd need the un-tuned base o3 model score for that.

In the meantime, comparing this tuned o3 model to other un-tuned base models is unfair (apples-to-oranges kind of comparison).

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by refulgentis on Friday December 20, 2024 @01:40PM (#42473597)
This emphasizes persons and a self-conceived victory narrative over the ground truth.

Models have regularly made progress on it, this is not new with the o-series.

Doing astoundingly well on it, and having a mutually shared PR interest with OpenAI in this instance, doesn't mean a pile of visual puzzles is actually AGI or some well thought out and designed benchmark of True Intelligence(tm). It's one type of visual puzzle.

I don't mean to be negative, but to inject a memento mori. Real story is some guys get together and ride off Chollet's name with some visual puzzles from ye olde IQ test, and the deal was Chollet then gets to show up and say it proves program synthesis is required for True Intelligence.

Getting this score is extremely impressive but I don't assign more signal to it than any other benchmark with some thought to it.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by stego-tech on Friday December 20, 2024 @04:18PM (#42475145)
I won't be as brutal in my wording, but I agree with the sentiment. This was something drilled into me as someone with a hobby in PC Gaming and Photography: benchmarks, while handy measures of potential capabilities, are not guarantees of real world performance. Very few PC gamers completely reinstall the OS before benchmarking to remove all potential cruft or performance impacts, just as very few photographers exclusively take photos of test materials.

While I appreciate the benchmark and its goals (not to mention the puzzles - I quite enjoy figuring them out), successfully passing this benchmark does not demonstrate or guarantee real world capabilities or performance. This is why I increasingly side-eye this field and its obsession with constantly passing benchmarks and then moving the goal posts to a newer, harder benchmark that claims to be a better simulation of human capabilities than the last one: it reeks of squandered capital and a lack of a viable/profitable product, at least to my sniff test. Rather than simply capitalize on their actual accomplishments (which LLMs are - natural language interaction is huge!), they're trying to prove to Capital that with a few (hundred) billion more in investments, they can make AGI out of this and replace all those expensive humans.

They've built the most advanced prediction engines ever conceived, and insist they're best used to replace labor. I'm not sure how they reached that conclusion, but considering even their own models refute this use case for LLMs, I doubt their execution ability on that lofty promise.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by modeless on Friday December 20, 2024 @01:48PM (#42473688)
Solving ARC doesn't mean we have AGI. Also o3 presumably isn't doing program synthesis, seemingly proving Francois wrong on that front. (Not sure I believe the speculation about o3's internals in the link.)

What I'm saying is the fact that as models are getting better at reasoning they are also scoring better on ARC proves that it is measuring something relating to reasoning. And nobody else has come up with a comparable benchmark that is so easy for humans and so hard for LLMs. Even today, let alone five years ago when ARC was released. ARC was visionary.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by hdjjhhvvhga on Friday December 20, 2024 @02:00PM (#42473808)
Your argumentation seems convincing but I'd like to offer a competitive narrative: any benchmark that is public becomes completely useless because companies optimize for it - especially AI that depends on piles of money and they need some proof they are developing.

That's why I have some private benchmarks and I'm sorry to say that the transition from GTP4 to o1 wasn't unambiguously a step forward (in some tasks yes, in some not).

On the other hand, private benchmarks are even less useful to the general public than the public ones, so we have to deal with what we have - but many of us just treat it as noise and don't give it much significance. Ultimately, the models should defend themselves by performing the tasks individual users want them to do.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by stonemetal12 on Friday December 20, 2024 @02:15PM (#42473961)
Rather any Logic puzzle you post on the internet as something AIs are bad at is in the next round of training data so AIs get better at that specific question. Not because AI companies are optimizing for a benchmark but because they suck up everything.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by modeless on Friday December 20, 2024 @02:24PM (#42474071)
ARC has two test sets that are not posted on the Internet. One is kept completely private and never shared. It is used when testing open source models and the models are run locally with no internet access. The other test set is used when testing closed source models that are only available as APIs. So it could be leaked in theory, but it is still not posted on the internet and can't be in any web crawls.

You could argue that the models can get an advantage by looking at the training set which is on the internet. But all of the tasks are unique and generalizing from the training set to the test set is the whole point of the benchmark. So it's not a serious objection.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by foobiekr on Friday December 20, 2024 @07:14PM (#42476419)
Given the delivery mechanism for OpenAI, how do they actually keep it private?
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by modeless on Friday December 20, 2024 @07:51PM (#42476607)
> So it could be leaked in theory

That's why they have two test sets. But OpenAI has legally committed to not training on data passed to the API. I don't believe OpenAI would burn their reputation and risk legal action just to cheat on ARC. And what they've reported is not implausible IMO.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by sensanaty on Saturday December 21, 2024 @07:20AM (#42479262)
Yeah I'm sure the Microsoft-backed company headed by Mr. Worldcoin Altman whose sole mission statement so far has been to overhype every single product they released wouldn't dare cheat on one of these benchmarks that "prove" AGI (as they've been claiming since GPT-2).
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by QuantumGood on Friday December 20, 2024 @02:04PM (#42473847)
Gaming the benchmarks usually needs to be considered first when evaluating new results.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by bubblyworld on Friday December 20, 2024 @02:23PM (#42474053)
I think gaming the benchmarks is encouraged in the ARC AGI context. If you look at the public test cases you'll see they test a ton of pretty abstract concepts - space, colour, basic laws of physics like gravity/magnetism, movement, identity and lots of other stuff (highly recommend exploring them). Getting an AI to do well at all, regardless of whether it was gamed or not, is the whole challenge!
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by chaps on Friday December 20, 2024 @02:14PM (#42473949)
Honestly, is gaming benchmarks actually a problem in this space in that it still shows something useful? Just means we need more benchmarks, yeah? It really feels not unlike keggle competitions.

We do the same exact stuff with real people with programming challenges and such where people just study common interview questions rather than learning the material holistically. And since we know that people game these interview type questions, we can adjust the interview processes to minimize gamification.... which itself leads to gamification and back to step one. That's not ideal an ideal feedback loop of course, but people still get jobs and churn out "productive work" out of it.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by ben_w on Friday December 20, 2024 @02:19PM (#42474013)
AI are very good at gaming benchmarks. Both as overfitting and as Goodhart's law, gaming benchmarks has been a core problem during training for as long as I've been interested in the field.

Sometimes this manifests as "outside the box thinking", like how a genetic algorithm got an "oscillator" which was really just an antenna.

It is a hard problem, and yes we still both need and can make more and better benchmarks; but it's still a problem because it means the benchmarks we do have are overstating competence.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by chaps on Friday December 20, 2024 @02:54PM (#42474378)
We're in agreement!

What's endlessly interesting to me with all of this is how surprisingly quick the benchmarking feedback loops have become plus the level of scrutiny each one receives. We (as a culture/society/whatever) don't really treat human benchmarking criteria with the same scrutiny such that feedback loops are useful and lead to productive changes to the benchmarking system itself. So from that POV it feels like substantial progress continues to be made through these benchmarks.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by CamperBob2 on Friday December 20, 2024 @02:38PM (#42474217)
The idea behind this particular benchmark, at least, is that it can't be gamed. What are some ways to game ARC-AGI, meaning to pass it without developing the required internal model and insights?

In principle you can't optimize specifically for ARC-AGI, train against it, or overfit to it, because only a few of the puzzles are publicly disclosed.

Whether it lives up to that goal, I don't know, but their approach sounded good when I first heard about it.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by psb217 on Friday December 20, 2024 @03:25PM (#42474674)
Well, with billions in funding you could task a hundred or so very well paid researchers to do their best at reverse engineering the general thought process which went into ARC-AGI, and then generate fresh training data and labeled CoTs until the numbers go up.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by CamperBob2 on Friday December 20, 2024 @03:29PM (#42474722)
Right, but the ARC-AGI people would counter by saying they're welcome to do just that. In doing so -- again in their view -- the researchers would create a model that could be considered capable of AGI.

I spent a couple of hours looking at the publicly-available puzzles, and was really impressed at how much room for creativity the format provides. Supposedly the puzzles are "easy for humans," but some of them were not... at least not for me.

(It did occur to me that a better test of AGI might be the ability to generate new, innovative ARC-AGI puzzles.)

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by psb217 on Friday December 20, 2024 @07:37PM (#42476547)
It's tricky to judge the difficulty of these sorts of things. Eg, breadth of possibilities isn't an automatic sign of difficulty. I imagine the space of programming problems permits as much variety as ARC-AGI, but since we're more familiar with problems presented as natural language descriptions of programming tasks, and since we know there's tons of relevant text on the web, we see the abstract pictographic ARC-AGI tasks as more novel, challenging, etc. But, to an LLM, any task we can conceive of will be (roughly) as familiar as the amount of relevant training data it's seen. It's legitimately hard to internalize this.

For a space of tasks which are well-suited to programmatic generation, as ARC-AGI is by design, if we can do a decent job of reverse engineering the underlying problem generating grammar, then we can make an LLM as familiar with the task as we're willing to spend on compute.

To be clear, I'm not saying solving these sorts of tasks is unimpressive. I'm saying that I find it unsuprising (in light of past results) and not that strong of a signal about further progress towards the singularity, or FOOM, or whatever. For any of these closed-ish domain tasks, I feel a bit like they're solving Go for the umpteenth time. We now know that if you collect enough relevant training data and train a big enough model with enough GPUs, the training loss will go down and you'll probably get solid performance on the test set. Trillions of reasonably diverse training tokens buys you a lot of generalization. Ie, supervised learning works. This is the horse Ilya Sutskever's ridden to many glorious victories and the big driver of OpenAI's success -- a firm belief that other folks were leaving A LOT of performance on the table due to a lack of belief in the power of their own inventions.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by refulgentis on Friday December 20, 2024 @02:34PM (#42474180)
> Solving ARC doesn't mean we have AGI. Also o3 presumably isn't doing program synthesis, seemingly proving Francois wrong on that front.

Agreed.

> And nobody else has come up with a comparable benchmark that is so easy for humans and so hard for LLMs.

? There's plenty.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by modeless on Friday December 20, 2024 @02:50PM (#42474336)
I'd love to hear about more. Which ones are you thinking of?
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by refulgentis on Friday December 20, 2024 @03:15PM (#42474585)
- "Are You Human" https://arxiv.org/pdf/2410.09569 is designed to be directly on target, i.e. cross cutting set of questions that are easy for humans, but challenging for LLMs, Instead of one type of visual puzzle. Much better than ARC for the purpose you're looking for.

- SimpleBench https://simple-bench.com/ (similar to above; great landing page w/scores that show human / ai gap)

- PIQA (physical question answering, i.e. "how do i get a yolk out of a water bottle", common favorite of local llm enthusiasts in /r/localllama https://paperswithcode.com/dataset/piqa

- Berkeley Function-Calling (I prefer https://gorilla.cs.berkeley.edu/leaderboard.html)

AI search googled "llm benchmarks challenging for ai easy for humans", and "language model benchmarks that humans excel at but ai struggles with", and "tasks that are easy for humans but difficult for natural language ai".

It also mentioned Moravec's Paradox is a known framing of this concept, started going down that rabbit hole because the resources were fascinating, but, had to hold back and submit this reply first. :)

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by modeless on Friday December 20, 2024 @04:26PM (#42475208)
Thanks for the pointers! I hadn't seen Are You Human. Looks like it's only two months old. Of course it is much easier to design a test specifically to thwart LLMs now that we have them. It seems to me that it is designed to exploit details of LLM structure like tokenizers (e.g. character counting tasks) rather than to provide any sort of general reasoning benchmark. As such it seems relatively straightforward to improve performance in ways that wouldn't necessarily represent progress in general reasoning. And today's LLMs are not nearly as far from human performance on the benchmark as they were on ARC for many years after it was released.

SimpleBench looks more interesting. Also less than two months old. It doesn't look as challenging for LLMs as ARC, since o1-preview and Sonnet 3.5 already got half of the human baseline score; they did much worse on ARC. But I like the direction!

PIQA is cool but not hard enough for LLMs.

I'm not sure Berkeley Function-Calling represents tasks that are "easy" for average humans. Maybe programmers could perform well on it. But I like ARC in part because the tasks do seem like they should be quite straightforward even for non-expert humans.

Moravec's paradox isn't a benchmark per se. I tend to believe that there is no real paradox and all we need is larger datasets to see the same scaling laws that we have for LLMs. I see good evidence in this direction: https://www.physicalintelligence.company/blog/pi0

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by refulgentis on Friday December 20, 2024 @06:03PM (#42475995)
> "I'm not sure Berkeley Function-Calling represents tasks that are easy for average humans. Maybe programmers could perform well on it."

Functions in this context are not programming function calls. In this context, function calls are a now-deprecated LLM API name for "parse input into this JSON template." No programmer experience needed. Entity extraction by another name, except, that'd be harder: here, you're told up front exactly the set of entities to identify. :)

> "Moravec's paradox isn't a benchmark per se."

Yup! It's a paradox :)

> "Of course it is much easier to design a test specifically to thwart LLMs now that we have them"

Yes.

Though, I'm concerned a simple yes might be insufficient for illumination here.

It is a tautology (it's easier to design a test that $X fails when you have access to $X), and it's unlikely you meant to just share a tautology.

A potential unstated-but-maybe-intended-communication is "it was hard to come up with ARC before LLMs existed" --- LLMs existed in 2019 :)

If they didn't, a hacky way to come up with a test that's hard for the top AIs at the time, BERT-era, would be to use one type of visual puzzle.

If, for conversations sake, we ignore that it is exactly one type of visual puzzle, and that it wasn't designed to be easy for humans, then we can engage with: "its the only one thats easy for humans, but hard for LLMs" --- this was demonstrated as untrue as well.

I don't think I have much to contribute past that, once we're at "It is a singular example of a benchmark thats easy for humans but nigh-impossible for llms, at least in 2019, and this required singular insight", there's just too much that's not even wrong, in the Pauli sense, and it's in a different universe from the original claims:

- "Congratulations to Francois Chollet on making the most interesting and challenging LLM benchmark so far."

- "A lot of people have criticized ARC as not being relevant or indicative of true reasoning...The fact that [o-series models show progress on ARC proves that what it measures really is relevant and important for reasoning."

- "...nobody could quantify exactly the ways the models were deficient..."

- "What we need right now are "easy" benchmarks that these models nevertheless fail."

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by CamperBob2 on Friday December 20, 2024 @04:58PM (#42475474)
How long has SimpleBench been posted? Out of the first 6 questions at https://simple-bench.com/try-yourself, o1-pro got 5/6 right.

It was interesting to see how it failed on question 6: https://chatgpt.com/c/6765e70e-44b0-800b-97bd-928919f04fbe

Apparently LLMs do not consider global thermonuclear war to be all that big a deal, for better or worse.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by Pannoniae on Friday December 20, 2024 @05:16PM (#42475619)
Don't worry, I also got that wrong :) I thought her affair would be the biggest problem for John.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by jquery on Friday December 20, 2024 @06:04PM (#42476000)
John was an ex, not her partner. Tricky.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by HarHarVeryFunny on Friday December 20, 2024 @07:42PM (#42476568)
> o3 presumably isn't doing program synthesis

I'd guess it's doing natural language procedural synthesis, the same way a human might (i.e. figuring the sequence of steps to effect the transformation), but it may well be doing (sub-)solution verification by using the procedural description to generate code whose output can then be compared to the provided examples.

While OpenAI haven't said exactly what the architecture of o1/o3 are, the gist of it is pretty clear - basically adding "tree" search and iteration on top of the underlying LLM, driven by some RL-based post-training that imparts generic problem solving biases to the model. Maybe there is a separate model orchestrating the search and solution evaluation.

I think there are many tasks that are easy enough for humans but hard/impossible for these models - the ultimate one in terms of commercial value would be to take an "off the shelf model" and treat it as an intern/apprentice and teach it to become competent in a entire job it was never trained on. Have it participate in team meetings and communications, and become a drop-in replacement for a human performing that job (any job that an be performed remotely without a physical presence).

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by danielmarkbruce on Friday December 20, 2024 @04:27PM (#42475211)
100%. The hype is misguided. I doubt half the people excited about the result have even looked at what the benchmark is.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by danielmarkbruce on Friday December 20, 2024 @04:25PM (#42475198)
Highly challenging for LLMs because it has nothing to do with language. LLMs and their training processes have all kinds of optimizations for language and how it's presented.

This benchmark has done a wonderful job with marketing by picking a great name. It's largely irrelevant for LLMs despite the fact it's difficult.

Consider how much of the model is just noise for a task like this given the low amount of information in each token and the high embedding dimensions used in LLMs.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by computerex on Friday December 20, 2024 @07:36PM (#42476537)
The benchmark is designed to test for AGI and intelligence, specifically the ability to solve novel problems.

If the hypothesis is that LLMs are the “computer” that drives the AGI then of course the benchmark is relevant in testing for AGI.

I don’t think you understand the benchmark and its motivation. ARC AGI benchmark problems are extremely easy and simple for humans. But LLMs fail spectacularly at them. Why they fail is irrelevant, the fact they fail though means that we don’t have AGI.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by danielmarkbruce on Friday December 20, 2024 @08:22PM (#42476759)
> The benchmark is designed to test for AGI and intelligence, specifically the ability to solve novel problems.

It's a bunch of visual puzzles. They aren't a test for AGI because it's not general. If models (or any other system for that matter) could solve it, we'd be saying "this is a stupid puzzle, it has no practical significance". It's a test of some sort of specific intelligence. On top of that, the vast majority of blind people would fail - are they not generally intelligent?

The name is marketing hype.

The benchmark could be called "random puzzles LLMs are not good at because they haven't been optimized for it because it's not valuable benchmark". Sure, it wasn't designed for LLMs, but throwing LLMs at it and saying "see?" is dumb. We can throw in benchmarks for tennis playing, chess playing, video game playing, car driving and a bajillion other things while we are at it.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by NateEag on Saturday December 21, 2024 @01:58AM (#42477975)
And all that is kind of irrelevant, because if LLMs were human-level general intelligence, they would solve all these questions correctly without blinking.

But they don't. Not even the best ones.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by pama on Saturday December 21, 2024 @09:08AM (#42479768)
No human would score high on that puzzle if the images were given to them as a series of tokens. Even previous LLMs scored much better than humans if tested in the same way.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by Tanjreeve on Sunday December 22, 2024 @04:47AM (#42485376)
And most humans would do well on maths problems if the input was given to them as binary. The reason that reversal isn't important is that the Tokens are an implementation detail for how an AI is meant to solve real world problems that humans face while noone cares about humans solving tokens.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by danielmarkbruce on Monday December 23, 2024 @08:48PM (#42499039)
Humans communicate with each other to get things done. We have to think carefully how we communicate with each other given the shortcomings of humans and shortcomings of different communication mediums.

The fact that we might need to be mindful of how we communicate with a person/system/whatever doesn't mean too much in the context of AI. Just like humans, the details of how they work will need to be considered, and the standard trope of "that's an implementation detail" won't work.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by adamgordonbell on Friday December 20, 2024 @04:32PM (#42475261)
There is a benchmark, NovelQA, that LLMs don't dominate when it feels like they should. The benchmark is to read a novel and answer questions about it.

LLMs are below human evaluation, as I last looked, but it doesn't get much attention.

Once it is passed, I'd like to see one that is solving the mystery in a mystery book right before it's revealed.

We'd need unpublished mystery novels to use for that benchmark, but I think it gets at what I think of as reasoning.

https://novelqa.github.io/

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by loxias on Friday December 20, 2024 @07:55PM (#42476628)
NovelQA is a great one! I also like GSM-Symbolic -- a benchmark based on making _symbolic templates_ of quite easy questions, and sampling them repeatedly, varying things like which proper nouns are used, what order relevant details appear, how many irrelevant details (GSM-NoOp) and where they are in the question, things like that.

LLMs are far, _far_ below human on elementary problems, once you allow any variation and stop spoonfeeding perfectly phrased word problems. :)

https://machinelearning.apple.com/research/gsm-symbolic

https://arxiv.org/pdf/2410.05229

Paper came out in October, I don't think many have fully absorbed the implications.

It's hard to take any of the claims of "LLMs can do reasoning!" seriously, once you understand that simply changing what names are used in a 8th grade math word problem can have dramatic impact on the accuracy.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by meta_x_ai on Friday December 20, 2024 @04:50PM (#42475414)
Looks like it's not updated for nearly a year and I'm guessing Gemini 2.0 Flash with 2m context will simply crush it
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by adamgordonbell on Friday December 20, 2024 @05:34PM (#42475771)
That's true. They don't have Claude 3.5 on there either. So maybe it's not relevant anymore, but I'm not sure.

If so, let's move on to the murder mysteries or more complex literary analysis.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by latency-guy2 on Friday December 20, 2024 @08:11PM (#42476706)
> I'd like to see one that is solving the mystery in a mystery book right before it's revealed.

I would think this is a not so good bench. Author does not write logically, they write for entertainment.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by adamgordonbell on Friday December 20, 2024 @08:23PM (#42476766)
So I'm thinking of something like Locked-room mystery where the idea is it's solvable, and the reader is given a chance to solve.

The reason it seems like an interesting bench, is it's a puzzle presented in a long context. Its like testing if an LLm is at Sherlock Holmes level of world and motivation modelling.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by rowanG077 on Friday December 20, 2024 @06:20PM (#42476115)
Benchmark how? Is it good if the LLM can or can't solve it?
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by CamperBob2 on Friday December 20, 2024 @04:37PM (#42475302)
Does it work on short stories, but not novels? If so, then that's just a minor question of context length that should self-resolve over time.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by adamgordonbell on Friday December 20, 2024 @04:40PM (#42475327)
The books fit in the current long context models, so it's not merely the context size constraint but the length is part of the issue, for sure.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by usaar333 on Saturday December 21, 2024 @02:09AM (#42478007)
That's an old leaderboard -- has no one checked any SOTA LLM in the last 8 months?
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by aimanbenbaha on Friday December 20, 2024 @06:39PM (#42476242)
Because LLMs are on an off-ramp path towards AGI. A generally intelligent system can brute force its way with just memory.

Once a model recognizes a weakness through reasoning with CoT when posed to a certain problem and gets the agency to adapt to solve that problem that's a precursor towards real AGI capability!

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by justanotherjoe on Saturday December 21, 2024 @12:03AM (#42477656)
i am confused cause this dataset is visual-based, and yet being used to measure 'LLM'. I feel like the visual nature of it was really the biggest hurdle to solving it.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by internet_points on Saturday December 21, 2024 @03:49AM (#42478318)
> The fact that scaled reasoning models are finally showing progress on ARC proves that what it measures really is relevant and important for reasoning.

One might also interpret that as "the fact that models which are studying to the test are getting better at the test" (Goodhart's law), not that they're actually reasoning.