News for hackers. Stuff that matters.
Re:NanoGPT Slowrun: Language Modeling with Limited Data, Inf... by magicalhippo on Thursday March 05, 2026 @10:08AM (#47262414)
Maybe some newer references are better, but my mind went to the Model Soups paper[1]:

The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-out validation set, discarding the remainder. In this paper, we revisit the second step of this procedure in the context of fine-tuning large pre-trained models, where fine-tuned models often appear to lie in a single low error basin. We show that averaging the weights of multiple models fine-tuned with different hyperparameter configurations often improves accuracy and robustness. Unlike a conventional ensemble, we may average many models without incurring any additional inference or memory costs -- we call the results "model soups."

[1]: https://arxiv.org/abs/2203.05482

There are no comments on this comment yet.