Has anyone here actually tried to use preemptible GPUs on GCE? You're lucky if you can get the VM to stay up for more than 10 minutes.

Litany of failure here: https://twitter.com/theshawwn/status/1174480402779648000

Perhaps I just got unlucky and the situation is improved now. But it was a waste of a day for me.

Disclosure: I work on Google Cloud.

Sorry about that. We (I) had screwed up and allowed users to smash themselves into the wall of "Huh, you keep getting preempted, but there's technically a slot right here". We've re-enabled some backpressure to keep you from getting hit over and over again, and instead see a stockout error when you should probably try another (less full) zone.

Hey, neat! Thanks for your work. I was hoping that this was the case, and I haven't tried since September. But I'll give it another shot now.

Feel free to send me an email (contact info in profile), if you want to go back and forth.

I killed a lot of time with this exact issue a couple months ago as well. Quite frustrating, and it was even worse trying to include the preemptible GPUs in a kubernetes cluster. It would often take so long to add the GPU to the cluster that it would get killed before it even ran any real workload. This cycle would repeat for an hour before a machine stayed up long enough to actually schedule kubernetes pods on them.

It would be nice of GCP didn't bill for pre-emptible instances that got system killed within 10 minutes of starting; or something reasonable like that.

I doubt it makes a material difference to their (or our) bottom line, but it's just a nice gesture and makes you never feel like "damn, I literally wasted money".

While this is pretty frustrating and seemed to be the result of a bug, I would personally only use preemptible instances are when I have everything scripted, and save the state of my work when you get a termination signal.

To me, they're not really sold as "cheaper, just not 100% guaranteed"; they're sold as "cheap, it will be killed at any moments notice so".

I haven't in awhile; when I used them more heavily, I got the full 24 hours. It's possible that it's region-dependent; IIRC I had better luck with us-east1 than us-central1.

I should test more at some point.

To clarify, does that mean you can continuously train a model for e.g. hundred of hours or are you still limited by the 24 hours notebook limit?

The page said "Longer running notebooks and fewer idle timeouts mean you disconnect less often." I'm guessing there is a limit but it has been increased for the pro customers. In contrast on GCE you can train for 22 hours if you want.

The page says its twice the traditional Cloud.

You can work around that by saving your model and reloading.

Can you elaborate on this?

Have model training make checkpoints every five minutes. When the training gets interrupted, just restart it on a new instance from the last checkpoint. Repeat as necessary until hundreds of hours of training is completed.

Yes, exactly. To add one more detail: mount your Google Drive and save the checkpoint there.

Cloud storage bucket works better than Google Drive for both training data and checkpoints; and if you're doing training on TPUs, then I believe that you must do the data exchange via cloud storage.

OK, that also works. Just don't rely solely on the storage that comes with your notebook instance.