News for hackers. Stuff that matters.
Re:Common misconceptions about the complexity in robotics vs... by dbspin on Saturday January 11, 2025 @04:54AM (#42664663)
I find it odd that the article doesn't address the apparent success of training with transformer based models in virtual environments to build models that are then mapped onto the real world. This is being used in everything from building datasets for self driving cars, to navigation and task completion for humanoid robots. Nvidia have their omniverse project [1], but there are countless other examples [2][3][4]. Isn't this obviously the way to build the corpus of experience needed to train these kinds of cross modal models?

[1] https://www.nvidia.com/en-us/industries/robotics/#:~:text=NV....

[2] https://www.sciencedirect.com/science/article/abs/pii/S00978...

[3] https://techcrunch.com/2024/01/04/google-outlines-new-method...

[4] https://techxplore.com/news/2024-09-google-deepmind-unveils-...

Re:Common misconceptions about the complexity in robotics vs... by cybernoodles on Saturday January 11, 2025 @05:14AM (#42664766)
A common practice is to train a transformer model to control a given robot model in simulation by first teleoperating the simulated model with some controller (keyboard, joystick, etc.) to complete the task and create a dataset, and then setting up the simulator to permute the environment variables such as frictions, textures, etc (domain randomization) and run many epochs at faster than real time until a final policy converges. If the right things were randomized and your demonstration examples provided enough variation of information, it should generalize well to the actual hardware.