News for hackers. Stuff that matters.
Re:SmolGPT: A minimal PyTorch implementation for training a ... by c0wb0yc0d3r on Wednesday January 29, 2025 @07:52PM (#42873470)
Can someone help me understand what I’m looking at here? This repository allows me to train a specific model on a specific data set, and finally test the result? Is that correct?

I am interested in how large and small language models are trained, but as someone who has little knowledge in this world I find it hard to cut through the noise to find useful information.

Really I’m looking for an open source project that helps a person gain this knowledge. Something like a docker container that encapsulates all the dependencies. When training it will use any available gpu or tell me why my gpu can’t be used and then fall back to cpu. Then had a simple interface to test the training results. Finally you can easily pull back the curtain to understand the process in better detail and maybe even adapt it to different model to experiment.

Does something like that exist?

Re:SmolGPT: A minimal PyTorch implementation for training a ... by timnetworks on Wednesday January 29, 2025 @08:14PM (#42873632)
As opposed to inference (like generating text and images), training requires some more math (fp16 or bf16) and a single CPU generally won't cut it.

The prepare/train/generate instructions in the github linked are pretty much it for the 'how' of training a model. You give it a task and it does it for 1 billion trillion epochs and saves the changes incrementally (or not).

Training a LoRA for an image model may be more approachable, there's more blog entries etc on this, and the process is largely similar, except you're doing it for a single slice instead of the whole network.

[edit] I'm also learning so correct me if I'm off, hn!

Re:SmolGPT: A minimal PyTorch implementation for training a ... by sva_ on Thursday January 30, 2025 @04:56AM (#42876418)
> You give it a task and it does it for 1 billion trillion epochs and saves the changes incrementally (or not).

Somewhat confusingly, big LLM are most just trained for 1 epoch afaik.

Re:SmolGPT: A minimal PyTorch implementation for training a ... by _joel on Thursday January 30, 2025 @05:55AM (#42876663)
I've seen 3 epochs on some of the finetuning R1 blog posts. It's not my field so not sure how valid that is.
Re:SmolGPT: A minimal PyTorch implementation for training a ... by sva_ on Saturday February 01, 2025 @05:13AM (#42897327)
Yeah, fine-tuning is different from pretraining
Re:SmolGPT: A minimal PyTorch implementation for training a ... by SJC_Hacker on Wednesday January 29, 2025 @09:58PM (#42874310)
Do you have a good theoretical foundation in ML ? You will also need some linear algebra.

If not would invest the time in a decent course, there are plenty online, even offline if you are close enough to where its offered. I took one from Andrew Ng on Coursera years ago, which used Matlab. There are probably much better, more up-to-date options now, especially now that LLMs are very in-vogue. The fundamentals such as gradient descent, ANNs and back-propagation however, is still relevant, and hasn't changed much.

Trying to understand what code is doing without that foundation will be an exercise in futility.

Re:SmolGPT: A minimal PyTorch implementation for training a ... by c0wb0yc0d3r on Thursday January 30, 2025 @12:31AM (#42875185)
I don’t have a solid ML foundation, and it’s been a decade or more since I’ve worked with linear algebra.

For now I think that might be too deep for what I’m after. I’m at the beach looking out at the vast ocean that is machine learning and LLMs.

Re:SmolGPT: A minimal PyTorch implementation for training a ... by barrenko on Thursday January 30, 2025 @03:30AM (#42876034)
You're probably having the right hunch, it takes a crapload of time, especially if you want to implement and not just "get an intuition".
Re:SmolGPT: A minimal PyTorch implementation for training a ... by MacTea on Wednesday January 29, 2025 @11:45PM (#42874947)
https://course.fast.ai/ is the best. From their site: " A free course designed for people with some coding experience, who want to learn how to apply deep learning and machine learning to practical problems. "
Re:SmolGPT: A minimal PyTorch implementation for training a ... by c0wb0yc0d3r on Thursday January 30, 2025 @12:44AM (#42875241)
This is at the top of my lunch time learning list. Not quite what I’ve been envisioning but it’s in the right direction. Thanks!
Re:SmolGPT: A minimal PyTorch implementation for training a ... by TheTaytay on Thursday January 30, 2025 @12:45PM (#42880102)
If you are looking for something that actually explains most concepts behind it, Karpathy’s series will teach you what you want to know. (Mentioned elsewhere) If you are looking for command line tools to fine tune and evaluate models on known datasets, this article is a good take!

https://www.philschmid.de/fine-tune-llms-in-2025