Re:The GPT Architecture, on a Napkin
by WanderPanda on Sunday December 11, 2022 @01:16PM
(#33945752)
Is treating the position embeddings as trainable weights already out of fashion again?
[ Parent
| on HN ]
according to the Attention is all you need paper there wasn't huge differences between using sin/cos and trainable weights. But the paper is a little bit old, so I don't know what is the current sota regarding positional embeddings
I’m new to this stuff, but as I understand it, the “Attention is all you need” paper stated that training the positional encoding weights didn’t improve results for language models specifically, but other papers found that vision transformers performed better with trainable weights.