I've heard from ML engineers at larger shops that reproducibility is key to working at scale. That's how you track down "this training regime went to shit because of something we changed" versus "this training regime went to shit because on of the GPUs training it is starting to fail".