Re:RWKV: Reinventing RNNs for the Transformer Era
by coppsilgold on Tuesday May 23, 2023 @11:53AM
(#36046136)
Softmax takes a vector of arbitrary values and converts it into probabilities such that all the elements of the vector add to 1. Transformers use this to decide on the fractions.
This could be seen as "soft attention" where in theory there would be few winners for every "attention head".
It's also possible that the only purpose the softmax actually serves (or at least major component of) is that of normalization. Without it, the variance in the internal layer dynamics between training samples would be fairly large (some training samples may have tokens that interact heavily, while others not). Making optimization problematic.
This could be seen as "soft attention" where in theory there would be few winners for every "attention head".
It's also possible that the only purpose the softmax actually serves (or at least major component of) is that of normalization. Without it, the variance in the internal layer dynamics between training samples would be fairly large (some training samples may have tokens that interact heavily, while others not). Making optimization problematic.