News for hackers. Stuff that matters.
Re:Genie 2: A large-scale foundation world model by suddenlybananas on Wednesday December 04, 2024 @12:48PM (#42319945)
This is really not the avenue for house-hold robots. Interacting with the actual physical world is very different from creating a video game.
Re:Genie 2: A large-scale foundation world model by sangnoir on Wednesday December 04, 2024 @01:03PM (#42320120)
> Interacting with the actual physical world is very different from creating a video game

The major difference being the former scales very poorly for generating training data compared to the latter. Genie 2 is not even a video game and has worse fidelity that video games, the upside is it probably scales even better than video games for generating training scenarios. If you want androids in teal life, Genie 2 (or similar systems) is how you bootstrap the agent AI. The training pipeline will be: raw video -> Genie 2 -> game engine with rules -> physical robot

Re:Genie 2: A large-scale foundation world model by youoy on Wednesday December 04, 2024 @02:12PM (#42320886)
> The training pipeline will be: raw video -> Genie 2 -> game engine with rules -> physical robot

One of those arrows is not like the others

Re:Genie 2: A large-scale foundation world model by sangnoir on Wednesday December 04, 2024 @03:32PM (#42321737)
The final step is an oversimplification: purpose-built simulator -> deconstructed robot on a lab workbench -> controlled space -> "real world" with constraints -> real world

Any model would have to succeed in one stage before it can proceed to the next one.

Re:Genie 2: A large-scale foundation world model by adverbly on Wednesday December 04, 2024 @03:49PM (#42321896)
At the risk of sounding repetitive, one of those arrows is not like the others.
Re:Genie 2: A large-scale foundation world model by sangnoir on Wednesday December 04, 2024 @04:07PM (#42322057)
...and?
Re:Genie 2: A large-scale foundation world model by mosdl on Wednesday December 04, 2024 @04:49PM (#42322400)
How does turning an image into a game help with robots? Robots don't need to guess what they can't see, they would have sensors to tell them exactly what is there (like a self driving car).
Re:Genie 2: A large-scale foundation world model by sangnoir on Wednesday December 04, 2024 @05:48PM (#42322908)
> Robots don't need to guess what they can't see, they would have sensors to tell them exactly what is there (like a self driving car).

Self driving vars have cameras as part of their sensor suite, and have models to make sense of sensor data. Video will help with perception and classification (understanding the world) with no agency needed. Game-playing will help with planning, execution, and evaluation. Both functions are necessary, and those that come after rely on earlier capabilities

Re:Genie 2: A large-scale foundation world model by Chilko on Wednesday December 04, 2024 @05:30PM (#42322777)
I have no expertise in this area, but my assumption is that this could help for a broader sort of object/world permanence for robots - e.g. if something is no longer visible to the robot's sensors (e.g. behind an obstacle, smoke, etc) then it could use a model based on this type of tech to maintain a short-term estimate of its surroundings even when operating blind.
Re:Genie 2: A large-scale foundation world model by falcor84 on Wednesday December 04, 2024 @11:17PM (#42324923)
To be able to plan ahead, robots do absolutely need to plan ahead (read: "guess" or even "imagine") what they might encounter before they sense it. In your self driving car example, for instance, it needs to come up with various scenarios for what might be around the corner ahead of a turn, and assign reasonable probabilities to these scenarios. I absolutely see how a system like this could help with it.

For example, let's say that the car is approaching an intersection, and suddenly sees a puddle on the road to the left getting brighter - a visual world model like this might extrapolate a scenario that the brightness is the result of a car moving towards the intersection assigning this some probability, and signing another probably to a scenario that it's just a flickering headlight, and the car would then decide whether and how much to slow down.

In this example there is a sensor, but it definitely doesn't tell the robot "exactly what is there", and while we could try to write rules about what it should do, the Bitter Lesson tells us it's better to just let it create its own model.

Re:Genie 2: A large-scale foundation world model by j_timberlake on Thursday December 05, 2024 @09:35PM (#42335632)
No, they actually did use a genie-like model to train robots on household chores.

Page 8 of the Genie 1 paper: https://arxiv.org/abs/2402.15391