Disclosure: I work on Google Cloud (and know the OpenAI folks).

Fwiw, I think 2500 "V100-days" here is an extrapolation of ~1 2048-TPUv3 day. So approximately $8/hr * 24 hours * 2048 => $400k, if you know, you could make use of that TPU Pod for much of the rest of the days of the year :).

But yes, this is still the realm of "Do you have millions of dollars of ML infrastructure" (if you want it quickly). I'm kind of hoping gwern et al. will use Colab to slowly train a variant of this for free :).

We've moved up in the world! These days we'd train on one of our TFRC TPU pods. Colab is a lot of trouble and better avoided...

(We actually can create a TPU-2048 pod. For a short while, anyway, before it preempts. But we wouldn't need to if we used lucidrain's efficient attention implementations I mention in my other comment.)

Oh, I missed that! Fwiw, I’m a big proponent of 1/4-pod slices anyway :).

Is there a way to guarantee a specific GPU for Colab, and also have it run in the background without the runtime disconnecting? Even Colab Pro doesn't seem to guarantee a specific resource without the disconnects.

Nope. You need to have it open in a browser tab and to keep it from disconnecting you run this script:

    function ConnectButton(){document.querySelector("#top-toolbar > colab-connect-button").shadowRoot.querySelector("#connect").click(); console.log("Connect pushed"); } setInterval(ConnectButton,60000);

:) You still get booted every 6 hour or so.

... for 64x64 px resolution.

This is truly a brute force approach.

But add an AI superresolution model after that and you're off to the races!

The problem with superresolution is that by design, they focus on details and textures for upscaling, and avoid adding in any large-scale structure. So if you train it separately (ie there is no backpropagation from the final highres 224px image through the superresolution model all the way back through iGPT), you cripple the original model.

There is only so much about objects and the world you can learn from 64px thumbnails, and they mention that this is probably a reason iGPT wins on the tiny images like CIFAR but loses to the semi-supervised CNNs on ImageNet: because the CNNs are compute-efficient enough that they can train at the standard 224px and see all of the details and structure that disappears at iGPT's 64px, and learn end-to-end.

Cost to train the model (which can be used for many images, and images can be upscaled using AI):

$10000 * 2500days / 5yr = $15000 hardware cost

200W * 2500day * (0.10 USD / Whr) = $1.2M in electricity

If electricity cost $0.1/Whr, a cup of tea would cost ~$10 just to run the kettle. You've overestimated the cost of electricity by roughly a thousand-fold.

Residential electricity prices in California are more like 0.19 USD/kWh, which is 0.00019 USD/Wh.

Found the Brit.

I wonder what the cost is when taking second order effects into the equation. Leverage hides the true cost, burying the reality further and further from the obvious.

I’ll have to do a writeup sometime explaining why these numbers aren’t as surprising as they seem. They’re also not as wasteful as they seem.

Roughly, the boxes would be turned on anyway. Might as well put them to work.

And yeah, it draws more power when in use (by a lot). But the data center probably isn’t paying a huge premium on top of what they would already pay for electricity.

So all that’s left is feeling generally queasy about using so much electricity, rather than a practical feeling of “this is bad because it costs a lot.”

Economies of scale have some dramatic effects at the high ends: I doubt the datacenter’s costs are anywhere near linear for the amount of Whr they’re consuming. And even if they are, it’s still a rounding error considering the profitability of GoogSoftBook.

At some point it becomes linear though, there is a base cost to electricity. Google builds datacenters near coal power plants specifically to get (literally) dirt cheap electricity.

This is true. But researchers are divorced from those realities. The day-to-day researchers almost never have to think like "If I run this, will it cost a ton?" Those kinds of agreements are usually made on a company basis, i.e. there's no cost to just leave a model running and see what happens. (Which is a huge advantage, by the way.)

Just a correction here: we build datacenters near large sources of renewable / hydro plants (I consider hydro a borderline case). We buy renewable credits for everything and are even trying to timeshift load now to match renewable generation peaks.

tl;dr: not coal plants :).

Is it just a coincidence that the Eemshaven (eu-west4) datacenter is close to a coal plant then?

I'm pretty sure we're going to close that plant soon though.

> Is it just a coincidence that the Eemshaven (eu-west4) datacenter is close to a coal plant then?

Presumably, since:

"Our data center in Eemshaven was the first to be powered by 100% renewable energy from day one."

https://www.google.com/about/datacenters/locations/eemshaven...

I appreciate them buying credits, but as nice as those windmills and solar panels are, when the sun doesn't shine, and the wind doesn't blow, what gets burned is coal. Anyway I suppose that's more a policy issue of The Netherlands, I think it's quite shameful we still have coal plants. Those credits hide the real cost of renewable energy.

I'm not sure - they say the data center is "the first Google datacenter to be powered by 100% renewable energy from day one."

https://www.blog.google/around-the-globe/google-europe/dutch...

And they seem pretty serious about removing non-renewables altogether.

https://storage.googleapis.com/gweb-sustainability.appspot.c...

Electricity rates are kWh, so I think it would be 120k for electricity, not 1.2 mil.

$1200, off by a factor of 1000.

$.1 for KWhr not When, so $1200 electricity.

Good use of units makes it easy to spot and fix the mistake.