News for hackers. Stuff that matters.
Re:The GPT Architecture, on a Napkin by cuuupid on Sunday December 11, 2022 @11:04AM (#33944315)
Note on the sinusoidal encoding, the reason it’s used is generally speaking twofold:

1 - To encode position somehow (which author details)

2 - Because sine is easy “noise” for the network to learn.

There’s also a bunch of cool tricks here even down to the PyTorch implementation to optimize this encoding by exploiting the nature of sine/cosine which is an added reason for its popularity in Transformer architectures. If you like math I recommend diving into it as it’s quick but fun!

(Side note it’s also falling out of fashion for other encoding methods. e.g. rotary positional encoding is vastly popular in the reformer branch of transformers)

Re:The GPT Architecture, on a Napkin by khazhoux on Monday December 12, 2022 @02:01AM (#33951773)
For a newbie in all this, what's the reason that position isn't just encoded as an integer?
Re:The GPT Architecture, on a Napkin by pnt12 on Monday December 12, 2022 @07:39AM (#33954036)
Not an expert, but took my masters in data science some years ago:

My interpretation is the sinusoidal converts the values from 0..N to 0..1. This may result in more predictable changes than using large integer numbers, such that the same word in different positions doesn't lose all its meaning.

Using integers and having a layer to compute this operation could also work, so maybe this is an optimization, eg it reduces training time or yields better results.

Re:The GPT Architecture, on a Napkin by WanderPanda on Sunday December 11, 2022 @01:16PM (#33945752)
Is treating the position embeddings as trainable weights already out of fashion again?
Re:The GPT Architecture, on a Napkin by alexmolas on Sunday December 11, 2022 @03:35PM (#33947206)
according to the Attention is all you need paper there wasn't huge differences between using sin/cos and trainable weights. But the paper is a little bit old, so I don't know what is the current sota regarding positional embeddings
Re:The GPT Architecture, on a Napkin by yunwal on Sunday December 11, 2022 @05:55PM (#33948522)
I’m new to this stuff, but as I understand it, the “Attention is all you need” paper stated that training the positional encoding weights didn’t improve results for language models specifically, but other papers found that vision transformers performed better with trainable weights.