The primary benefit of transformers is that they generally can reference things within the context, but RNNs have trouble with that. The trade off is that transformers have memory and compute requirements quadratic in the context length.