News for hackers. Stuff that matters.
Re:How to train large deep learning models as a startup by etrain on Thursday October 07, 2021 @11:25AM (#28787548)
Check out Determined https://github.com/determined-ai/determined to help manage this kind of work at scale: Determined leverages Horovod under the hood, automatically manages cloud resources and can get you up on spot instances, T4's, etc. and will work on your local cluster as well. Gives you additional features like experiment management, scheduling, profiling, model registry, advanced hyperparameter tuning, etc.

Full disclosure: I'm a founder of the project.

Re:How to train large deep learning models as a startup by bigbillheck on Thursday October 07, 2021 @07:27PM (#28793482)
Oh hey I interviewed with y'all a few years back, glad to see you're still around.
Re:How to train large deep learning models as a startup by dylanbfox on Thursday October 07, 2021 @11:27AM (#28787576)
Interesting. How do you guys manage spot interruptions when training on spot instances?
Re:How to train large deep learning models as a startup by etrain on Thursday October 07, 2021 @02:49PM (#28790569)
Users expose their model to our Trial API (https://docs.determined.ai/latest/topic-guides/model-definit...), the base class then implements a training loop (which can be enhanced with user-supplied callbacks, metrics, etc.) that has a whole bunch of bells and whistles. Easy distributed (multi-GPU and multi-node) training, automatic checkpointing, fault tolerance, etc.

Concretely, the system is regularly taking checkpoints (which include model weights and optimizer state) and so if the spots disappear (as they do), the system has enough information to resume from where things were last checkpointed when resources become available again.

Re:How to train large deep learning models as a startup by PickleAI on Thursday October 07, 2021 @11:56AM (#28788007)
Thanks for going open source!