News for hackers. Stuff that matters.
Re:SmolGPT: A minimal PyTorch implementation for training a ... by timnetworks on Wednesday January 29, 2025 @08:14PM (#42873632)
As opposed to inference (like generating text and images), training requires some more math (fp16 or bf16) and a single CPU generally won't cut it.

The prepare/train/generate instructions in the github linked are pretty much it for the 'how' of training a model. You give it a task and it does it for 1 billion trillion epochs and saves the changes incrementally (or not).

Training a LoRA for an image model may be more approachable, there's more blog entries etc on this, and the process is largely similar, except you're doing it for a single slice instead of the whole network.

[edit] I'm also learning so correct me if I'm off, hn!

Re:SmolGPT: A minimal PyTorch implementation for training a ... by sva_ on Thursday January 30, 2025 @04:56AM (#42876418)
> You give it a task and it does it for 1 billion trillion epochs and saves the changes incrementally (or not).

Somewhat confusingly, big LLM are most just trained for 1 epoch afaik.

Re:SmolGPT: A minimal PyTorch implementation for training a ... by _joel on Thursday January 30, 2025 @05:55AM (#42876663)
I've seen 3 epochs on some of the finetuning R1 blog posts. It's not my field so not sure how valid that is.
Re:SmolGPT: A minimal PyTorch implementation for training a ... by sva_ on Saturday February 01, 2025 @05:13AM (#42897327)
Yeah, fine-tuning is different from pretraining