Re:Revealing example of self-attention, the building block o...
by jostmey on Saturday April 29, 2023 @09:22PM
(#35758881)
I just checked and the attention matrix is 768x768. Basically, I reshape each sample to be a single step with an embed size of 768. I hope this makes sense. I may rewrite the README later tonight to make this point clear.
Let me know if you think I still don't got it!!!
Re:Revealing example of self-attention, the building block o...
by GaggiX on Saturday April 29, 2023 @09:26PM
(#35758895)
If you have a single embedded vector, it should be clear that the self-attention matrix cannot be an arbitrary NxN but only a 1x1, follow the fix I have added to my previous comment.
Also if K, Q, V and the self-attention matrix have the same dimensions even when the embed_d and sequence dim have different sizes there is something wrong.
For a simple fix just make the attention matrix W [num_steps, num_steps] instead of [num_inputs, num_inputs] then "y_i = torch.matmul(x_v_i, w_i)" would be "y_i = torch.matmul(w_i, x_v_i)" and should be correct.