News for hackers. Stuff that matters.

A comment on Revealing example of self-attention, the building block of transformer AI models. Parent comment

Re:Revealing example of self-attention, the building block o... by cshimmin on Sunday April 30, 2023 @02:40PM (#35765352)
The dot product of two d_k-dimensional vectors is a scalar. So the entire NxN matrix qk_ij is just a bunch of scalar numbers. The rows are used to create a weighted sum (i.e. a linear combination) of the d_v-dimensional vectors.
[ Parent | on HN ]

There are no comments on this comment yet.