I'm curious about this comment and it being the top comment. Is this something people don't know? Everything in this paper was introduced in Attention Is All You Need[0]. They introduced Dot Product Attention, which is what everyone just refers to now as Attention, and they talk about the decoder and encoder framework. The encoder is just self attention `softmax(<q(x),k(x)>)v(x)` and decoder includes joint attention `softmax(<q(x),k(x)>)v(y)`

I have a lot of complaints about this paper because it only covers topics addressed in the main attention paper (Vaswani) and I can't see how it accomplishes anything but pulling citations away from grad students who did survey papers on Attention, which are more precise and have more coverage of the field. As a quick search, here's a survey paper from last year that has more in depth discussion and more mathematical precision[1].

Maybe because this is an area of research adjacent to my active area, but I'm a little confused at the attention (pun intended) it is getting. Seems just like Deep Mind's name is the major motivation.

[0] https://arxiv.org/abs/1706.03762

[1] https://arxiv.org/abs/2106.04554