Yeah thinking about it more, it does seem possible.

Like the attention masking done in transformers, I get why, but I also never understood why you wouldn't want attention to happen, at least about concepts and abstract ideas about what is coming up in the phrase before predicting a word