News for hackers. Stuff that matters.
Re:Genie 2: A large-scale foundation world model by golol on Wednesday December 04, 2024 @12:44PM (#42319904)
Do you want household androids? Because this kind of stuff is on the level of research a bery large step towards that. Think as it as ab example where we can make a model understand a lot of physical common sense stuff, which is the goal for robotics right now.
Re:Genie 2: A large-scale foundation world model by JTyQZSnP3cQGa8B on Wednesday December 04, 2024 @02:20PM (#42320987)
I don't understand how that is relevant. I certainly would not want household androids unless I'm completely disabled.
Re:Genie 2: A large-scale foundation world model by theshackleford on Wednesday December 04, 2024 @05:29PM (#42322767)
> I certainly would not want household androids unless I'm completely disabled.

That's nice. I'm not completely disabled, but I am disabled, and I very much would appreciate them, as my capability to do things over the longer term is very much not going to go in the direction of improving. As it is, there are a lot of things I now rely on people for, that at one time, I did not.

Whilst I recognise its probably not going to happen in a time span that is useful to me, I do wish it could, so that I could be less of a burden on those around me, and maintain a relative level of independence.

Re:Genie 2: A large-scale foundation world model by furyofantares on Thursday December 05, 2024 @01:10AM (#42325429)
A lot of people are disabled right now!

Unless you have a young/quick death, there's a really good chance you will be, too.

Re:Genie 2: A large-scale foundation world model by throwaway2037 on Thursday December 05, 2024 @03:01AM (#42325998)
No trolling: You wouldn't want robots to mow your lawn, maybe do the boring bits of cooking (prep! endless stirring!), clean your house, wash your clothes? Man, sign me up!
Re:Genie 2: A large-scale foundation world model by suddenlybananas on Wednesday December 04, 2024 @12:48PM (#42319945)
This is really not the avenue for house-hold robots. Interacting with the actual physical world is very different from creating a video game.
Re:Genie 2: A large-scale foundation world model by sangnoir on Wednesday December 04, 2024 @01:03PM (#42320120)
> Interacting with the actual physical world is very different from creating a video game

The major difference being the former scales very poorly for generating training data compared to the latter. Genie 2 is not even a video game and has worse fidelity that video games, the upside is it probably scales even better than video games for generating training scenarios. If you want androids in teal life, Genie 2 (or similar systems) is how you bootstrap the agent AI. The training pipeline will be: raw video -> Genie 2 -> game engine with rules -> physical robot

Re:Genie 2: A large-scale foundation world model by youoy on Wednesday December 04, 2024 @02:12PM (#42320886)
> The training pipeline will be: raw video -> Genie 2 -> game engine with rules -> physical robot

One of those arrows is not like the others

Re:Genie 2: A large-scale foundation world model by sangnoir on Wednesday December 04, 2024 @03:32PM (#42321737)
The final step is an oversimplification: purpose-built simulator -> deconstructed robot on a lab workbench -> controlled space -> "real world" with constraints -> real world

Any model would have to succeed in one stage before it can proceed to the next one.

Re:Genie 2: A large-scale foundation world model by adverbly on Wednesday December 04, 2024 @03:49PM (#42321896)
At the risk of sounding repetitive, one of those arrows is not like the others.
Re:Genie 2: A large-scale foundation world model by sangnoir on Wednesday December 04, 2024 @04:07PM (#42322057)
...and?
Re:Genie 2: A large-scale foundation world model by mosdl on Wednesday December 04, 2024 @04:49PM (#42322400)
How does turning an image into a game help with robots? Robots don't need to guess what they can't see, they would have sensors to tell them exactly what is there (like a self driving car).
Re:Genie 2: A large-scale foundation world model by sangnoir on Wednesday December 04, 2024 @05:48PM (#42322908)
> Robots don't need to guess what they can't see, they would have sensors to tell them exactly what is there (like a self driving car).

Self driving vars have cameras as part of their sensor suite, and have models to make sense of sensor data. Video will help with perception and classification (understanding the world) with no agency needed. Game-playing will help with planning, execution, and evaluation. Both functions are necessary, and those that come after rely on earlier capabilities

Re:Genie 2: A large-scale foundation world model by Chilko on Wednesday December 04, 2024 @05:30PM (#42322777)
I have no expertise in this area, but my assumption is that this could help for a broader sort of object/world permanence for robots - e.g. if something is no longer visible to the robot's sensors (e.g. behind an obstacle, smoke, etc) then it could use a model based on this type of tech to maintain a short-term estimate of its surroundings even when operating blind.
Re:Genie 2: A large-scale foundation world model by falcor84 on Wednesday December 04, 2024 @11:17PM (#42324923)
To be able to plan ahead, robots do absolutely need to plan ahead (read: "guess" or even "imagine") what they might encounter before they sense it. In your self driving car example, for instance, it needs to come up with various scenarios for what might be around the corner ahead of a turn, and assign reasonable probabilities to these scenarios. I absolutely see how a system like this could help with it.

For example, let's say that the car is approaching an intersection, and suddenly sees a puddle on the road to the left getting brighter - a visual world model like this might extrapolate a scenario that the brightness is the result of a car moving towards the intersection assigning this some probability, and signing another probably to a scenario that it's just a flickering headlight, and the car would then decide whether and how much to slow down.

In this example there is a sensor, but it definitely doesn't tell the robot "exactly what is there", and while we could try to write rules about what it should do, the Bitter Lesson tells us it's better to just let it create its own model.

Re:Genie 2: A large-scale foundation world model by j_timberlake on Thursday December 05, 2024 @09:35PM (#42335632)
No, they actually did use a genie-like model to train robots on household chores.

Page 8 of the Genie 1 paper: https://arxiv.org/abs/2402.15391