News for hackers. Stuff that matters.
Re:SmolGPT: A minimal PyTorch implementation for training a ... by sva_ on Thursday January 30, 2025 @04:56AM (#42876418)
> You give it a task and it does it for 1 billion trillion epochs and saves the changes incrementally (or not).

Somewhat confusingly, big LLM are most just trained for 1 epoch afaik.

Re:SmolGPT: A minimal PyTorch implementation for training a ... by _joel on Thursday January 30, 2025 @05:55AM (#42876663)
I've seen 3 epochs on some of the finetuning R1 blog posts. It's not my field so not sure how valid that is.
Re:SmolGPT: A minimal PyTorch implementation for training a ... by sva_ on Saturday February 01, 2025 @05:13AM (#42897327)
Yeah, fine-tuning is different from pretraining