The Illustrated Transformer is pretty great. I was pretty hazy after reading the paper back in 2017 and this resource helped a lot.

https://jalammar.github.io/illustrated-transformer/

Thanks, that one is really useful.

Excellent. Thanks for sharing.

Complicated from whose perspective? I don't go around commenting on systems programming articles about how low-level memory management and currency algorithms are too complicated, or commenting on category theory articles that the terminology is too obtuse and monads are too hard.

I agree that there probably could be a better "on ramp" into this material than "take an undergraduate linear algebra course", but ultimately it is a mathematical model and you're going to have to deal with the math at some point if you want to actually understand what's going on. Linear algebra and calculus are entry-level table stakes for understanding how machine learning works, and there's really no way around that.

I am absolutely certain it is possible to explain this stuff without using jargon and mathematical notation that is impenetrable to the majority of professional software engineers.

I agree, but that doesn't mean that every article on the subject needs to be written for that audience in that language.

At some point, at some level, you really do need to just learn what a dot product is and what a matrix is. It's not weird notation or jargon, these are fundamental concepts.

Just like if you want to really learn how programs work you can't refuse any explanation that talks about "variables" or "functions" because that's jargon.

You can explain it, but it's going to be more at the level of "the network looks at the words" type explanation.

The idea of the transformer somehow being a trainable key-value store is kind of abstract and weird and has little to do with the mathematics of it. The math part of that is how the dot product encodes for similarity between vectors, but beyond that it really is a "if you get it you get it" kind of thing.

Andrej Karpathy ie [0]?

[0] https://www.youtube.com/watch?v=kCc8FmEb1nY

Vector, matrix, weighted sum, and dot product are good places to start. In fact, these concepts are so useful that they are good places to start pretty much no matter where you want to go. 3D graphics, statistics, physics, ... and neural networks.

I found this article on “transformers from scratch”[0] to be a perfect (for me) middle ground between high level hand-wavy explanations and overly technical in-the-weeds academic or code treatments.

[0] https://e2eml.school/transformers.html

Pretty sure the current state is simply one of confused exploration … hopefully soon to be synthesized and simplified.

Agree. Each new "advance" seems like just another trick/quirk (and adds jargon). Still waiting for the real advance.

It took me a long time to wrap my head around the whole "key/query/value" explanation (and to be honest I regularly forget which vector is which). I find the "weighted sum of vectors" explanation much simpler/more intuitive; this blog post is IMO the best on the subject:

https://peterbloem.nl/blog/transformers