Reinforcement Learning as One Big Sequence Modeling Problem https://trajectory-transformer.github.io/
That said, I doubt we’ll be able to make an RL GAN, so maybe this is the best way. Though now I wonder how well an RL GAN might work…
You can use transformers as part of GANs [1], and you can even use them as discriminative models for images [2].
[1] https://arxiv.org/abs/2102.07074 [2] https://arxiv.org/abs/2010.11929
Reframing reinforcement learning as sequence modeling /should/ make it possible to reuse (m)any of the approaches we use for audio, including GANs. Generative audio is nicely analogous to RL problems : There's complex state and interesting predictive distributions, which shift subtly over time, and you need to combine short-term good behavior (good individual samples) in a reasonable way to get good long-term behavior (matching melspectra).
https://arxiv.org/abs/2106.01345
'We consider the following shift in paradigm: instead of training a policy through conventional RL algorithms like temporal difference (TD) learning [6], we will train transformer models on collected experience using a sequence modeling objective. This will allow us to bypass the need for bootstrapping for long term credit assignment – thereby avoiding one of the “deadly triad” [6] known to destabilize RL. It also avoids the need for discounting future rewards, as typically done in TD learning, which can induce undesirable short-sighted behaviors. Additionally, we can make use of existing transformer frameworks widely used in language and vision that are easy to scale, utilizing a large body of work studying stable training of transformer models.'