I think you are conflating "Transformers" and "autoregressive models". Transformers are a general purpose architecture for transforming sequences into other sequences with self-attention. AR models / GANs are frameworks for generative modeling. The model architecture is almost entirely orthogonal to the generative framework.

You can use transformers as part of GANs [1], and you can even use them as discriminative models for images [2].

[1] https://arxiv.org/abs/2102.07074 [2] https://arxiv.org/abs/2010.11929

Yeah, I work on on-device audio, so definitely agree on the expense problem. In audio we've now got a few different approaches that work really well for sequence modeling, and we're constantly finding cool ways to make inference run faster.

Reframing reinforcement learning as sequence modeling /should/ make it possible to reuse (m)any of the approaches we use for audio, including GANs. Generative audio is nicely analogous to RL problems : There's complex state and interesting predictive distributions, which shift subtly over time, and you need to combine short-term good behavior (good individual samples) in a reasonable way to get good long-term behavior (matching melspectra).

Aren't actor critic algorithms very close to GANs already? You have a generator/actor/policy that produces data and a discriminator/critic/q that says if the data is good or bad. The critic trains on the data generated by the actor and some extra info given by the user (rewards or example data) and the actor learns from the signal given by the critic.