News for hackers. Stuff that matters.
Re:The GPT Architecture, on a Napkin by khazhoux on Monday December 12, 2022 @02:01AM (#33951773)
For a newbie in all this, what's the reason that position isn't just encoded as an integer?
Re:The GPT Architecture, on a Napkin by pnt12 on Monday December 12, 2022 @07:39AM (#33954036)
Not an expert, but took my masters in data science some years ago:

My interpretation is the sinusoidal converts the values from 0..N to 0..1. This may result in more predictable changes than using large integer numbers, such that the same word in different positions doesn't lose all its meaning.

Using integers and having a layer to compute this operation could also work, so maybe this is an optimization, eg it reduces training time or yields better results.