Re:Revealing example of self-attention, the building block o...
by cshimmin on Sunday April 30, 2023 @02:40PM
(#35765352)
The dot product of two d_k-dimensional vectors is a scalar. So the entire NxN matrix qk_ij is just a bunch of scalar numbers. The rows are used to create a weighted sum (i.e. a linear combination) of the d_v-dimensional vectors.
> So, to compute the new information to add to item i, we take the weighted sum of the values v_j that item i liked
I don't get this part. If the v_k are d_v dimensional vectors, and if the input items i are all d_f dimensional vectors, then how are you "adding" these values back to the inputs when d_f != d_v?
The linked script doesn't seem to do any adding like this - instead, they take the output values and pass them through a linear layer, presumably throwing away the inputs. But your comment hints at the existence of a more residual sort of approach.