News for hackers. Stuff that matters.

A comment on Mistral 7B, the most powerful language model for its size to date, Apache 2.0. Parent comment

Re:Mistral 7B, the most powerful language model for its size... by Scene_Cast2 on Wednesday September 27, 2023 @03:24PM (#37679903)
Yep. Even if the initial seed for parameter init, the example shuffling seed, etc were constant, the distributed nature of training (and syncing the model across machines) would kill reproducibility. Not to mention resuming from checkpoints after gradient explosions, etc.
[ Parent | on HN ]
Re:Mistral 7B, the most powerful language model for its size... by monocasa on Wednesday September 27, 2023 @03:32PM (#37680031)
I've heard from ML engineers at larger shops that reproducibility is key to working at scale. That's how you track down "this training regime went to shit because of something we changed" versus "this training regime went to shit because on of the GPUs training it is starting to fail".
[ Parent | on HN ]