That fundamentally misunderstands the point. If it costs me $XXX amounts of dollars to compute the job I'm trying to replace, which only actually costs $X dollar, then it's not financially viable to use the model to replace the job. It makes no difference what the global percentage of computation that makes up.
Just like not optimizing the 1% bottleneck in your code when there's a 50% bottleneck- focus on the big players where the cheap, easy wins are first.
The paper explicitly says: """Important work by scholars at the University of Massachusetts Amherst allows us to understand the economic cost and carbon emissions implied by this computational burden. The answers are grim: Training such a model would cost US $100 billion and would produce as much carbon emissions as New York City does in a month. And if we estimate the computational burden of a 1 percent error rate, the results are considerably worse."""
(that paper was debunked by people who build and operate extremely large scale cloud machine learning systems).
Remember, Google already built and runs its TPU fleet continuously and the machines are almost always busy at at least 50% of their capacity, meaning the money is already being spent and the carbon (which is much smaller than their estimates) being spewed.
As for the rest of the paper, their arguments about the need to increase parameters to get better results are fairly simplistic, and filled with misleading information/untrue statements.
Basically, everything they claim in the article has been shown to be empirically not true. And the community is already grappling with the "too many parameters approach". The authors would have served the community far better by writing a less critical paper that focused on the opportunities to identify good approaches to parameter and compute reduction that don't affect performance. In looking at the main author's area of research, it looks like he is more of a economist/public policy wonk than DL expert, which I think negatively affects the quality of the paper.
Articles are often talking about something that won't end up mattering. But if you can't actually address the point the article is making on its own terms, you're going to struggle to refute it.
When I refute a paper I start with the weakest points; if I convince my audience to ignore the paper on those terms, I don't spend time refuting the main point (perhaps this is not a great technique, but it is fairly efficient). I didn't even really address their main arguments because after reading the paper, I was just aghast at some of the things that say.
Let's give an example like this: """The first part is true of all statistical models: To improve performance by a factor of k, at least k**2 more data points must be used to train the model."""
I guess that's true of nearly all modern statistical models, at least small ones with simple model functions and fairly small number of data points. But I don't think that most advanced deep learning experts think in those terms; modern DLs do not behave anything like classical statistical models. I think those experts see increasing the data as providing an opportunity for overparameterized systems to generalize in ways we don't undertand and don't follow normal statistical rules. Modern DL systems are more like complex systems with emergent properties, than statistical models as understand by modern statisticians.
Here's another example: they sort of ignore the fact that we only need to train a small number of really big models and then all other models are fine-tuned from that. To get a world-class tardigrade detector, I took mobilenet, gave it a few hundred extra examples, trained for less than an hour on my home GPU (a 3080Ti), and then re-used that model for millions of predictions on my microscope. I didn't have to retrain any model or use absurd amounts of extra data. I took advantage of all the work the original model trainign did to discover basis functions that can compactly encode the difference between tardigrade, algae, and dirt. I see a direct linear increase in my model's performance as I move to linearly larger models, and I need to add linear number of images to train more classes.
Similarly, for people doing molecular dynamics- the big CPU waster before DL- many parts of MD can now be approximated with DLs that are cheaper to run than MD, using models that were just trained once.
What about AlphaFold? Even if it cost DeepMind $100M in training time (and probably more in salaries), it has already generated some results that simply couldn't be produced without their technology- it didn't even exist! What they demonstrated, quite convincingly, is that algorithmic improvements could extract far more information from fairly cheap and low quality sequence data, compared to expensive structural data. So instead of extremely expensive MD sims or whatever to predict structure, you just run this model. My friends in pharma research (I work in pharma) are delighted with the results.
In short, I think the author's economic model is naive, I think his udnerstanding of improvement in DL is naive, and he undercounts the value of having a small number of huge models which are trained on O(nlogn), not n**2 data sets.
If DL costs are unsustainable, it means those other things (SAP, web hosting, and all the ridiculous stuff people waste cycles on) are even more unsustainable, and should be addressed first.