News for hackers. Stuff that matters.
Re:Building an LLM from Scratch: Automatic Differentiation by itissid on Thursday February 15, 2024 @05:48PM (#39390345)
Every one should go through this rite of passage work and get to the "Attention is all you need" implementation. It's a world where engineering and the academic papers are very close and reproducible and a must for you to progress in the field.

(see also andre karpathys zero to hero nn series on youtube as well its very good and similar to this work)

Re:Building an LLM from Scratch: Automatic Differentiation by calebkaiser on Thursday February 15, 2024 @08:44PM (#39392031)
I would also recommend going through Callum McDougall/Neel Nanda's fantastic Transformer from Scratch tutorial. It takes a different approach to conceptualizing the model (or at least, it implements it in a way which emphasizes different characteristics of Transformers and self-attention), which I found deeply satisfying when I first explored them.

https://arena-ch1-transformers.streamlit.app/%5B1.1%5D_Trans...

Re:Building an LLM from Scratch: Automatic Differentiation by joshua11 on Friday February 16, 2024 @01:41AM (#39393950)
Thanks for sharing. This is a nice resource
Re:Building an LLM from Scratch: Automatic Differentiation by bschne on Thursday February 15, 2024 @07:05PM (#39391142)
+1 for Karpathy, the series is really good
Re:Building an LLM from Scratch: Automatic Differentiation by wredue on Thursday February 15, 2024 @09:21PM (#39392337)
Is this YouTube series also “from scratch (but not really)”

Edit - it is. Not to talk down on the series. I’m sure it’s good, but it is actually “LLM with PyTorch”.

Edit - I looked again and I was actually not correct. He does ultimately use frameworks, but gives some early talk about how those function under the hood.

Re:Building an LLM from Scratch: Automatic Differentiation by deadfast on Friday February 16, 2024 @07:57AM (#39396336)
I appreciate you coming back and giving more details, it encourages me to look into it now. Maybe my expectations on the internet are just low, but I thought it was a virtuous act worth the effort, I wish more people would continue with skepticism but be willing to follow through and let their opinions change given solid evidence.
Re:Building an LLM from Scratch: Automatic Differentiation by simfoo on Friday February 16, 2024 @03:28AM (#39394549)
That magic moment in Karpathys first video when he gets to the loss function and calls backward for the first time - this is when it clicked for me. Highly recommended!