Re:You could have designed state of the art positional encoding
by alok-g on Monday November 18, 2024 @05:53PM
(#42178008)
On a related note, one thing I still do not understand is why are positional encodings 'added' to the token embeddings as opposed to (having a smaller position encoding vector that is) 'concatenated'. It would be great if someone could explain.
[ Parent
| on HN ]
Re:You could have designed state of the art positional encoding
by d3m0t3p on Monday November 18, 2024 @06:26PM
(#42178343)
Increasing the dimension causes a lot more computation, this is one of the main reason. You can see evidence of this in the multi head where the dim is reduced via a linear projection.
h_i = attention(W_i^Q Q^T @ W_i^K K) W_i^v V
h = W_o @ concat(h_1...h_8)
Re:You could have designed state of the art positional encoding
by bfelbo on Tuesday November 19, 2024 @02:18AM
(#42180788)
How many dimensions would you need to increase by to capture positional information?
Seems to me like itβd be a quite low number compared to the dimensionality of the semantic vectors?