related online discussion when this article came out:

from Eleuther: https://twitter.com/AiEleuther/status/1648782486736969728?s=...

from AI anon: https://twitter.com/nearcyan/status/1662937711156625408?s=20

Stella Biderman (coauthor and now lead of the Eleuther Foundation, Stella if you're reading this pls come on our pod sometime!!!!) https://twitter.com/BlancheMinerva/status/164878587828883456...

I'm quite busy right now, but maybe in late September or October?

Yes! I have a 5 pages doc of prep notes I made, happy to share :)

Please do! Thanks

Sorry for the criticism here, but your linked paper / article does nothing to explain the math behind transformers. You should re-name it something like 'Scaling Transformer Mathematics' or 'The Math Behind Scaling GPUs / Transformers' or whatever more descriptive name there is to describe what you're outlining in your article. 'Transformer Math 101' should at least explain 1) input embeddings 2) (keys, queries, values) and the idea behind the matrix multiplication of these linear / numeric sets of values 3) the Softmax computation in the original paper as well as the relevant matrix transformations which take place 4) dot products ( and how they relate to similarity scoring) 5) feed-forward neural networks and the gradient descent / backpropagation and how they work. There are many many concepts you didn't even touch upon. This is not 'Transformer Math 101' by any means.

fwiw i'm not the author of that doc, we just interviewed him, and the hn submission i linked to was also renamed presumably for similar concerns. we do have an "Algorithms 101" episode (in the theme of our Datasets 101 and Benchmarks 101 episode) where we have at least some of your topics lined up