We've moved up in the world! These days we'd train on one of our TFRC TPU pods. Colab is a lot of trouble and better avoided...
(We actually can create a TPU-2048 pod. For a short while, anyway, before it preempts. But we wouldn't need to if we used lucidrain's efficient attention implementations I mention in my other comment.)
Oh, I missed that! Fwiw, I’m a big proponent of 1/4-pod slices anyway :).
Re:Image GPT
by forgingahead on Wednesday June 17, 2020 @10:05PM
(#23559053)
Is there a way to guarantee a specific GPU for Colab, and also have it run in the background without the runtime disconnecting? Even Colab Pro doesn't seem to guarantee a specific resource without the disconnects.
Nope. You need to have it open in a browser tab and to keep it from disconnecting you run this script:
function ConnectButton(){document.querySelector("#top-toolbar > colab-connect-button").shadowRoot.querySelector("#connect").click(); console.log("Connect pushed"); } setInterval(ConnectButton,60000);
Fwiw, I think 2500 "V100-days" here is an extrapolation of ~1 2048-TPUv3 day. So approximately $8/hr * 24 hours * 2048 => $400k, if you know, you could make use of that TPU Pod for much of the rest of the days of the year :).
But yes, this is still the realm of "Do you have millions of dollars of ML infrastructure" (if you want it quickly). I'm kind of hoping gwern et al. will use Colab to slowly train a variant of this for free :).