News for hackers. Stuff that matters.
Re:A Visual Guide to Vision Transformers by causal on Tuesday April 16, 2024 @12:40PM (#40054170)
I like this, but think there is some crucial motivation missing in steps 10.1-10.3 regarding what query/key weights are and why they're needed.
Re:A Visual Guide to Vision Transformers by vikiomega9 on Tuesday April 16, 2024 @04:28PM (#40056762)
this post made sense to me https://teltam.github.io/posts/soft-dictionary-keys.html

It helps to think of kqv as a form of look up.

Re:A Visual Guide to Vision Transformers by ThouYS on Tuesday April 16, 2024 @01:06PM (#40054518)
yes, same issue in all transformer tutorials
Re:A Visual Guide to Vision Transformers by lordswork on Tuesday April 16, 2024 @03:24PM (#40056116)
I suspect this is because most people (including people writing these tutorials) don't have a strong grasp on this piece as well.
Re:A Visual Guide to Vision Transformers by causal on Tuesday April 16, 2024 @02:05PM (#40055231)
The 2b1b video was the first to make it click for me
Re:A Visual Guide to Vision Transformers by hotdogscout on Tuesday April 16, 2024 @03:18PM (#40056049)
You mean 3b1b (three blue one brown)?
Re:A Visual Guide to Vision Transformers by causal on Tuesday April 16, 2024 @03:43PM (#40056322)
Ah that's right, miscounted the blues
Re:A Visual Guide to Vision Transformers by abhgh on Tuesday April 16, 2024 @09:17PM (#40059358)
They are like "continuous" databases. See slides 4-5 here [1] - this is from a talk I had given a while ago.

[1] https://drive.google.com/file/d/12uHo9QIfS-jBpVTs3lmQ3BEpxhD...