News for hackers. Stuff that matters.
Re:Reasoning Models Reason Well, Until They Don't by PxldLtd on Friday October 31, 2025 @07:34AM (#45770879)
I think a good test of this seems to be to provide an image and get the model to predict what will happen next/if x occurs. They fail spectacularly at Rube-Goldberg machines. I think developing some sort of dedicated prediction model would help massively in extrapolating data. The human subconscious is filled with all sorts of parabolic prediction, gravity, momentum and various other fast-thinking paths that embed these calculations.
Re:Reasoning Models Reason Well, Until They Don't by yanis_t on Friday October 31, 2025 @07:46AM (#45770967)
Any example of that? One would think that predicting what comes next from an image is basically video generation, which works not perfect, but works somehow (Veo/Sora/Grok)
Re:Reasoning Models Reason Well, Until They Don't by PxldLtd on Friday October 31, 2025 @08:02AM (#45771083)
Here's one I made in Veo3.1 since gemini is the only premium AI I have access to.

Using this image - https://www.whimsicalwidgets.com/wp-content/uploads/2023/07/... and the prompt: "Generate a video demonstrating what will happen when a ball rolls down the top left ramp in this scene."

You'll see it struggles - https://streamable.com/5doxh2 , which is often the case with video gen. You have to describe carefully and orchestrate natural feeling motion and interactions.

You're welcome to try with any other models but I suspect very similar results.

Re:Reasoning Models Reason Well, Until They Don't by chamomeal on Friday October 31, 2025 @08:12AM (#45771168)
I love how it still copies the slow pan and zoom from rube goldberg machine videos, but it's just following along with utter nonsense lol
Re:Reasoning Models Reason Well, Until They Don't by galaxyLogic on Friday October 31, 2025 @03:44PM (#45775925)
A Goldbergs machine was not part of their training data. For humans, we have seem such things.
Re:Reasoning Models Reason Well, Until They Don't by autoexec on Friday October 31, 2025 @03:55PM (#45776030)
physics textbooks are though so it should know how they'd work, or at least know that balls don't spontaneously appear and disappear and that gears don't work when they aren't connected
Re:Reasoning Models Reason Well, Until They Don't by mannykannot on Friday October 31, 2025 @09:01AM (#45771523)
It is video generation, but succeeding at this task involves detailed reasoning about cause and effect to construct chains of events, and may not be something that can be readily completed by applying "intuitions" gained from "watching" lots of typical movies, where most of the events are stereotypical.
Re:Reasoning Models Reason Well, Until They Don't by pfortuny on Friday October 31, 2025 @09:05AM (#45771555)
Most amazing is asking any of the models to draw an 11-sided polygon and number the edges.
Re:Reasoning Models Reason Well, Until They Don't by Torkel on Friday October 31, 2025 @09:21AM (#45771707)
I asked gpt5, and it worked really well with a correct result. Did you expect it to fail?
Re:Reasoning Models Reason Well, Until They Don't by pfortuny on Saturday November 01, 2025 @03:04PM (#45784360)
It has failed me several times already, drawing at most an octagon or a 12-gon: I mean create an image, not a program to do it.
Re:Reasoning Models Reason Well, Until They Don't by dns_snek on Monday November 03, 2025 @04:28AM (#45797295)
It drew a 10-sided shape, numbering the vertices instead of the edges, and the labels were wrong (some numbers repeated, others skipped)