Re:Building an LLM from Scratch: Automatic Differentiation
by calebkaiser on Thursday February 15, 2024 @08:44PM
(#39392031)
I would also recommend going through Callum McDougall/Neel Nanda's fantastic Transformer from Scratch tutorial. It takes a different approach to conceptualizing the model (or at least, it implements it in a way which emphasizes different characteristics of Transformers and self-attention), which I found deeply satisfying when I first explored them.
https://arena-ch1-transformers.streamlit.app/%5B1.1%5D_Trans...