Have model training make checkpoints every five minutes. When the training gets interrupted, just restart it on a new instance from the last checkpoint. Repeat as necessary until hundreds of hours of training is completed.

Yes, exactly. To add one more detail: mount your Google Drive and save the checkpoint there.

Cloud storage bucket works better than Google Drive for both training data and checkpoints; and if you're doing training on TPUs, then I believe that you must do the data exchange via cloud storage.

OK, that also works. Just don't rely solely on the storage that comes with your notebook instance.