News for hackers. Stuff that matters.
Re:SmolGPT: A minimal PyTorch implementation for training a ... by SJC_Hacker on Wednesday January 29, 2025 @09:58PM (#42874310)
Do you have a good theoretical foundation in ML ? You will also need some linear algebra.

If not would invest the time in a decent course, there are plenty online, even offline if you are close enough to where its offered. I took one from Andrew Ng on Coursera years ago, which used Matlab. There are probably much better, more up-to-date options now, especially now that LLMs are very in-vogue. The fundamentals such as gradient descent, ANNs and back-propagation however, is still relevant, and hasn't changed much.

Trying to understand what code is doing without that foundation will be an exercise in futility.

Re:SmolGPT: A minimal PyTorch implementation for training a ... by c0wb0yc0d3r on Thursday January 30, 2025 @12:31AM (#42875185)
I don’t have a solid ML foundation, and it’s been a decade or more since I’ve worked with linear algebra.

For now I think that might be too deep for what I’m after. I’m at the beach looking out at the vast ocean that is machine learning and LLMs.

Re:SmolGPT: A minimal PyTorch implementation for training a ... by barrenko on Thursday January 30, 2025 @03:30AM (#42876034)
You're probably having the right hunch, it takes a crapload of time, especially if you want to implement and not just "get an intuition".