Re:What is RT-2? Google DeepMind’s vision-language-action mo...
by Xeophon on Friday July 28, 2023 @11:10AM
(#36907930)
Related work is the planning paper by Valmeekan et al [1]. The gist is that LLMs are incapable of planning, which is due to their autoregressive nature. METAs Head of AI Yann Lecun also talks about this topic in a talk [2]. As RT2 is based on a similar architecture, I think the results will be similar.
[1] https://arxiv.org/abs/2305.15771 [2] https://youtu.be/x10964w00zk
P.S.: I also was told that the key here is that in automated planning you can't have a human in the loop doing the actual learning. If you are going to prompt engineer or get a human in the loop to a degree that you effectively fool yourself that the robot is solving a problem then its not planning.
[1] https://en.wikipedia.org/wiki/Sussman_anomaly [2] https://chat.openai.com/share/16a8a0e9-7422-41da-a192-6393cc... [3] https://twitter.com/rao2z/status/1599462959788744704?s=20