Yuck. I hate how expensive transformers are. You can see it clearly in transformers vs gans; GANs can generate a frame in about 20ms, whereas it takes seconds or more to make a frame with transformers. It’s not even clear that it’s necessarily better quality.

That said, I doubt we’ll be able to make an RL GAN, so maybe this is the best way. Though now I wonder how well an RL GAN might work…

I think you are conflating "Transformers" and "autoregressive models". Transformers are a general purpose architecture for transforming sequences into other sequences with self-attention. AR models / GANs are frameworks for generative modeling. The model architecture is almost entirely orthogonal to the generative framework.

You can use transformers as part of GANs [1], and you can even use them as discriminative models for images [2].

[1] https://arxiv.org/abs/2102.07074 [2] https://arxiv.org/abs/2010.11929

Yeah, I work on on-device audio, so definitely agree on the expense problem. In audio we've now got a few different approaches that work really well for sequence modeling, and we're constantly finding cool ways to make inference run faster.

Reframing reinforcement learning as sequence modeling /should/ make it possible to reuse (m)any of the approaches we use for audio, including GANs. Generative audio is nicely analogous to RL problems : There's complex state and interesting predictive distributions, which shift subtly over time, and you need to combine short-term good behavior (good individual samples) in a reasonable way to get good long-term behavior (matching melspectra).

Aren't actor critic algorithms very close to GANs already? You have a generator/actor/policy that produces data and a discriminator/critic/q that says if the data is good or bad. The critic trains on the data generated by the actor and some extra info given by the user (rewards or example data) and the actor learns from the signal given by the critic.