NVIDIA's Isaac Gym project revealed GPU's capability of performing massively parallel simulation for gym-style environments. Detailed information can be found in the following paper:

[1] Makoviychuk, Viktor, et al. "Isaac Gym: High-Performance GPU Based Physics Simulation For Robot Learning." Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). 2021.

At its release, people commented on Twitter that "it is the MNIST moment for reinforcement learning." And over the past year, I saw several follow-up works and tested NVIDIA's implementations.

For example, a demo by this blog

https://towardsdatascience.com/a-new-era-of-massively-parall...

The question is, does that technique help advance Reinforcement Learning, as expected?

It's not new. A paper in 2021 [0] showed you can train a quadruped robot to walk in minutes using parallel simulation in GPU, and then deploy it on the physical robot. Being parallel it is faster, and more so if on GPU. But sim-to-reality transfer is still a concern, and the architecture doesn't help if there are sparse rewards.

[0] https://arxiv.org/abs/2109.11978?context=cs.LG

Thanks for the information and sharing the paper. I also read it. Agree with you that the "simulation-to-reality gap" would be more critical to the reinforcement learning community.