Re:Copy is all you need
by lolinder on Monday July 17, 2023 @11:11AM
(#36759209)
I agree that the copycats are wearing thin, but the original paper's title seems fine to me. It's an accurate description of the breakthrough they made. The first few sentences of the Attention is All You Need abstract explain it pretty well:
> The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
Re:Copy is all you need
by macleginn on Monday July 17, 2023 @04:40PM
(#36763749)
The first sentence seems to imply that the main thing they want to do away with is the encoder–decoder arch, which they actually kept. (But BERT and GPT later did manage to simplify it.)
> The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
https://proceedings.neurips.cc/paper/2017/file/3f5ee243547de...