Re:Deep Learning’s Diminishing Returns – Cost of improvement...
by dekhn on Friday June 16, 2023 @06:19PM
(#36364733)
The paper advances several lines of arguments. Environmental is one of them, it's used as a negative consequence of their purported scaling laws for training.
The paper explicitly says: """Important work by scholars at the University of Massachusetts Amherst allows us to understand the economic cost and carbon emissions implied by this computational burden. The answers are grim: Training such a model would cost US $100 billion and would produce as much carbon emissions as New York City does in a month. And if we estimate the computational burden of a 1 percent error rate, the results are considerably worse."""
(that paper was debunked by people who build and operate extremely large scale cloud machine learning systems).
Remember, Google already built and runs its TPU fleet continuously and the machines are almost always busy at at least 50% of their capacity, meaning the money is already being spent and the carbon (which is much smaller than their estimates) being spewed.
As for the rest of the paper, their arguments about the need to increase parameters to get better results are fairly simplistic, and filled with misleading information/untrue statements.
Basically, everything they claim in the article has been shown to be empirically not true. And the community is already grappling with the "too many parameters approach". The authors would have served the community far better by writing a less critical paper that focused on the opportunities to identify good approaches to parameter and compute reduction that don't affect performance. In looking at the main author's area of research, it looks like he is more of a economist/public policy wonk than DL expert, which I think negatively affects the quality of the paper.
The paper explicitly says: """Important work by scholars at the University of Massachusetts Amherst allows us to understand the economic cost and carbon emissions implied by this computational burden. The answers are grim: Training such a model would cost US $100 billion and would produce as much carbon emissions as New York City does in a month. And if we estimate the computational burden of a 1 percent error rate, the results are considerably worse."""
(that paper was debunked by people who build and operate extremely large scale cloud machine learning systems).
Remember, Google already built and runs its TPU fleet continuously and the machines are almost always busy at at least 50% of their capacity, meaning the money is already being spent and the carbon (which is much smaller than their estimates) being spewed.
As for the rest of the paper, their arguments about the need to increase parameters to get better results are fairly simplistic, and filled with misleading information/untrue statements.
Basically, everything they claim in the article has been shown to be empirically not true. And the community is already grappling with the "too many parameters approach". The authors would have served the community far better by writing a less critical paper that focused on the opportunities to identify good approaches to parameter and compute reduction that don't affect performance. In looking at the main author's area of research, it looks like he is more of a economist/public policy wonk than DL expert, which I think negatively affects the quality of the paper.