Re:You could have designed state of the art positional encoding
by FL33TW00D on Monday November 18, 2024 @08:25AM
(#42172097)
Hm! Very interesting! Thank you for taking the time to debug that.
I'm going to have to think hard about how to rewrite the motivating example to explain this best.
Edit: updated the post, thanks for pointing out the pernicious init values!
Further, the attention weights are uniform and identical for both sentences:
That is not expected. It's because the Linear layers are initialised with such small values. And the softmax causes a collapse.Trying random weights on a larger scale: