Re:Colab Pro
by PeterisP on Sunday February 09, 2020 @05:59PM
(#22284924)
Have model training make checkpoints every five minutes. When the training gets interrupted, just restart it on a new instance from the last checkpoint. Repeat as necessary until hundreds of hours of training is completed.
[ Parent
| on HN ]
Yes, exactly. To add one more detail: mount your Google Drive and save the checkpoint there.
Cloud storage bucket works better than Google Drive for both training data and checkpoints; and if you're doing training on TPUs, then I believe that you must do the data exchange via cloud storage.