News for hackers. Stuff that matters.
Re:What is RT-2? Google DeepMind’s vision-language-action mo... by itissid on Friday July 28, 2023 @10:47AM (#36907581)
It would be interesting to see how it solve AI Planning Problems like BlocksWorld[1]. I've read that in the past with these things[2][3] when multiple goals needed to be met at once and there was interaction between them, it just falls over itself. Being able to generate coherent plans and execute them, as I understand, an important aspect of generating Action Sequences given a State and thus for planning in robotics. How are these overcome in RT2?

P.S.: I also was told that the key here is that in automated planning you can't have a human in the loop doing the actual learning. If you are going to prompt engineer or get a human in the loop to a degree that you effectively fool yourself that the robot is solving a problem then its not planning.

[1] https://en.wikipedia.org/wiki/Sussman_anomaly [2] https://chat.openai.com/share/16a8a0e9-7422-41da-a192-6393cc... [3] https://twitter.com/rao2z/status/1599462959788744704?s=20

Re:What is RT-2? Google DeepMind’s vision-language-action mo... by Xeophon on Friday July 28, 2023 @11:10AM (#36907930)
Related work is the planning paper by Valmeekan et al [1]. The gist is that LLMs are incapable of planning, which is due to their autoregressive nature. METAs Head of AI Yann Lecun also talks about this topic in a talk [2]. As RT2 is based on a similar architecture, I think the results will be similar.

[1] https://arxiv.org/abs/2305.15771 [2] https://youtu.be/x10964w00zk

Re:What is RT-2? Google DeepMind’s vision-language-action mo... by itissid on Friday July 28, 2023 @11:23AM (#36908205)
That arxiv link to the paper is also the same guy I mentioned in my twitter reference. Subbarao :)