News for hackers. Stuff that matters.
Re:Revealing example of self-attention, the building block o... by GaggiX on Saturday April 29, 2023 @09:26PM (#35758895)
If you have a single embedded vector, it should be clear that the self-attention matrix cannot be an arbitrary NxN but only a 1x1, follow the fix I have added to my previous comment.

Also if K, Q, V and the self-attention matrix have the same dimensions even when the embed_d and sequence dim have different sizes there is something wrong.

Re:Revealing example of self-attention, the building block o... by jostmey on Sunday April 30, 2023 @12:48AM (#35759759)
Okay, I think it is fixed. I appreciate your help. If you see this, let me know if the issue still exists!
Re:Revealing example of self-attention, the building block o... by GaggiX on Sunday April 30, 2023 @01:00AM (#35759809)
Now it should be correct.