News for hackers. Stuff that matters.
Re:The GPT Architecture, on a Napkin by alexmolas on Sunday December 11, 2022 @03:35PM (#33947206)
according to the Attention is all you need paper there wasn't huge differences between using sin/cos and trainable weights. But the paper is a little bit old, so I don't know what is the current sota regarding positional embeddings
Re:The GPT Architecture, on a Napkin by yunwal on Sunday December 11, 2022 @05:55PM (#33948522)
I’m new to this stuff, but as I understand it, the “Attention is all you need” paper stated that training the positional encoding weights didn’t improve results for language models specifically, but other papers found that vision transformers performed better with trainable weights.