> what's the proposal there?

That's what Google was afraid of: the alternative is to decide that the cost of the model isn't worth the effort (externalities < benefit) and to abandon it. Which might be the correct decision, but one that loses Google a lot of potential money.

The paper didn't attempt to examine if the benefits of the model outweighed the costs. They calculated the total energy used and called it a day.

Correct. It left it up to the model creators to potentially evaluate those tradeoffs for future models.

I find the conclusions ludicrous. Should we also abandon particle accelerators because they use too much electrity for ill-defined benefits?

Really the claim is toxic academia style knife fighting for tenure at best.

I bet all the big tech companies spend way more electricity on things other than training big models.

"You use too much electricity" is tantamount to "tech is a flawed business model". Arguably solvable with increased green energy sources but that's more a job for society in general than just tech companies.

How often is the electricity use considered a constraint in algorithm design for AI models?

Moore's law isn't the only way to gain efficiency, and even if the available compute goes up, doesn't mean the total power usage will go down

Do less AI training, so less AI could be a valid alternative.

Part of the issue in these events is that the paper that was written by Google's own ethics researchers simply repeats estimates of AI energy consumption from outside academic papers, instead of examining them. The reality is that Google's ML infrastructure is dramatically more energy efficient than the rest of the industry. A person with access to these facts who chooses to ignore them isn't really doing their best as a researcher, they're just grinding an ax.

There are other examples against the meme of AI using "a lot of electricity", for example a recent paper from Google where they show that a learned computational fluid dynamics technique uses only 2% the energy of standard techniques.

https://arxiv.org/pdf/2102.01010.pdf

Not only are ML techniques finding ways to do physics with less energy, the amount of waste by non-ML computing absolutely swamps ML computing. I don't care if training a model uses the amount of CO2 that a translatlanic flight uses, because those flights are flying constantly, some of them not filled to the max. The whole criticism about energy in the paper seemed to miss major details that I'd expect top researchers to get right.

BTW, I ran a large scale computing platform at Google for several years. It made absolutely enormous, stunningly huge amounts of CPU available to scientists and we used it to make several important breakthroughs. However, after a few years, I finally was shown what the power cost implications were, and I shut down the project because the costs were not justified by the scientific benefits. Compared to what I was doing, ML is relatively skimpy on CPU.

I'm not sure that your scientific facilities cumulatively spent as much CPU time as DeepMind spent playing StarCraft. If we want to talk about energy and ethics in machine learning I guess that's where I would start.

Yes, my calculations used far more time than DM, with the caveat that my jobs ran on CPUs while most of their training is on accelerators.

I believe we reported that Exacycle provided 700K high speed Xeon cores for over a year, and the actual number is far higher (I can't share it because you'd be able to make a reasonable estimate on how many computers Google has).

> I'm not sure that your scientific facilities cumulatively spent as much CPU time as DeepMind spent playing StarCraft.

You gotta pay if you wanna play. Humans are the result of a much longer and more expensive evolutionary process. It's only normal to have to pay a little to do the same for AI.

The proposal is not to continue the trend of simply adding titanic amounts of parameters to models and develop more efficient ones.