We're also realising we don't need full transformers all the time. Token mixing might suffice, there are quite a few papers replacing the core of the transformer with something simpler.