Re:Building an LLM from Scratch: Automatic Differentiation
by calebkaiser on Thursday February 15, 2024 @08:44PM
(#39392031)
I would also recommend going through Callum McDougall/Neel Nanda's fantastic Transformer from Scratch tutorial. It takes a different approach to conceptualizing the model (or at least, it implements it in a way which emphasizes different characteristics of Transformers and self-attention), which I found deeply satisfying when I first explored them.
https://arena-ch1-transformers.streamlit.app/%5B1.1%5D_Trans...
Re:Building an LLM from Scratch: Automatic Differentiation
by joshua11 on Friday February 16, 2024 @01:41AM
(#39393950)
Thanks for sharing. This is a nice resource
Re:Building an LLM from Scratch: Automatic Differentiation
by bschne on Thursday February 15, 2024 @07:05PM
(#39391142)
+1 for Karpathy, the series is really good
Re:Building an LLM from Scratch: Automatic Differentiation
by wredue on Thursday February 15, 2024 @09:21PM
(#39392337)
Is this YouTube series also “from scratch (but not really)”
Edit - it is. Not to talk down on the series. I’m sure it’s good, but it is actually “LLM with PyTorch”.
Edit - I looked again and I was actually not correct. He does ultimately use frameworks, but gives some early talk about how those function under the hood.
Re:Building an LLM from Scratch: Automatic Differentiation
by deadfast on Friday February 16, 2024 @07:57AM
(#39396336)
I appreciate you coming back and giving more details, it encourages me to look into it now. Maybe my expectations on the internet are just low, but I thought it was a virtuous act worth the effort, I wish more people would continue with skepticism but be willing to follow through and let their opinions change given solid evidence.
(see also andre karpathys zero to hero nn series on youtube as well its very good and similar to this work)