Re:How to train large deep learning models as a startup
by dylanbfox on Thursday October 07, 2021 @11:27AM
(#28787576)
Interesting. How do you guys manage spot interruptions when training on spot instances?
[ Parent
| on HN ]
Re:How to train large deep learning models as a startup
by etrain on Thursday October 07, 2021 @02:49PM
(#28790569)
Users expose their model to our Trial API (https://docs.determined.ai/latest/topic-guides/model-definit...), the base class then implements a training loop (which can be enhanced with user-supplied callbacks, metrics, etc.) that has a whole bunch of bells and whistles. Easy distributed (multi-GPU and multi-node) training, automatic checkpointing, fault tolerance, etc.
Concretely, the system is regularly taking checkpoints (which include model weights and optimizer state) and so if the spots disappear (as they do), the system has enough information to resume from where things were last checkpointed when resources become available again.