Re:Colab Pro
by ampdepolymerase on Saturday February 08, 2020 @04:43PM
(#22278614)
To clarify, does that mean you can continuously train a model for e.g. hundred of hours or are you still limited by the 24 hours notebook limit?
[ Parent
| on HN ]
The page said "Longer running notebooks and fewer idle timeouts mean you disconnect less often." I'm guessing there is a limit but it has been increased for the pro customers. In contrast on GCE you can train for 22 hours if you want.
Re:Colab Pro
by piadodjanho on Saturday February 08, 2020 @11:19PM
(#22280376)
The page says its twice the traditional Cloud.
You can work around that by saving your model and reloading.
Re:Colab Pro
by ampdepolymerase on Sunday February 09, 2020 @11:53AM
(#22282988)
Can you elaborate on this?
Have model training make checkpoints every five minutes. When the training gets interrupted, just restart it on a new instance from the last checkpoint. Repeat as necessary until hundreds of hours of training is completed.
Yes, exactly. To add one more detail: mount your Google Drive and save the checkpoint there.
Cloud storage bucket works better than Google Drive for both training data and checkpoints; and if you're doing training on TPUs, then I believe that you must do the data exchange via cloud storage.