Re:You could have designed state of the art positional encoding
by bfelbo on Tuesday November 19, 2024 @02:18AM
(#42180788)
How many dimensions would you need to increase by to capture positional information?
Seems to me like itβd be a quite low number compared to the dimensionality of the semantic vectors?
h_i = attention(W_i^Q Q^T @ W_i^K K) W_i^v V
h = W_o @ concat(h_1...h_8)