News for hackers. Stuff that matters.
Re:Revealing example of self-attention, the building block o... by jostmey on Saturday April 29, 2023 @09:22PM (#35758881)
I just checked and the attention matrix is 768x768. Basically, I reshape each sample to be a single step with an embed size of 768. I hope this makes sense. I may rewrite the README later tonight to make this point clear.

Let me know if you think I still don't got it!!!

Re:Revealing example of self-attention, the building block o... by GaggiX on Saturday April 29, 2023 @09:26PM (#35758895)
If you have a single embedded vector, it should be clear that the self-attention matrix cannot be an arbitrary NxN but only a 1x1, follow the fix I have added to my previous comment.

Also if K, Q, V and the self-attention matrix have the same dimensions even when the embed_d and sequence dim have different sizes there is something wrong.

Re:Revealing example of self-attention, the building block o... by jostmey on Sunday April 30, 2023 @12:48AM (#35759759)
Okay, I think it is fixed. I appreciate your help. If you see this, let me know if the issue still exists!
Re:Revealing example of self-attention, the building block o... by GaggiX on Sunday April 30, 2023 @01:00AM (#35759809)
Now it should be correct.