News for hackers. Stuff that matters.
Re:The GPT Architecture, on a Napkin by pnt12 on Monday December 12, 2022 @07:39AM (#33954036)
Not an expert, but took my masters in data science some years ago:

My interpretation is the sinusoidal converts the values from 0..N to 0..1. This may result in more predictable changes than using large integer numbers, such that the same word in different positions doesn't lose all its meaning.

Using integers and having a layer to compute this operation could also work, so maybe this is an optimization, eg it reduces training time or yields better results.

There are no comments on this comment yet.