Re:It takes a lot of energy for machines to learn
by _jal on Wednesday December 30, 2020 @05:03PM
(#25586700)
I think it is reasonable to assume that any advances in efficiency will be subsumed by increased use, much in the same way increased energy efficiency doesn't decrease absolute overall use.
Re:It takes a lot of energy for machines to learn
by joe_the_user on Wednesday December 30, 2020 @08:05PM
(#25588359)
ML hardware is no doubt progressing. Whether we can expect a rapid rate is a different matter. The various Moore's Laws have been breaking down (chip speed stopped doubling a while back). GPU are the sort of chip that could most benefit from just transistors doubling (last Mooore's law standing) but even this Moore's Law is breaking down.
ML model size has had it's own Moore's Law, with standard model growing exponentially in size [1]. And this implies model are going butt up against the limits even more than they have already. Whether the "bitter lesson"[2] of ML is true inherently is up for a question. That current researchers have accepted it seems given.
[1] https://openai.com/blog/ai-and-compute/
[2] http://www.incompleteideas.net/IncIdeas/BitterLesson.html
The energy efficiency of machine learning hardware is progressing at a rapid rate. It's not accurate to assume that the energy costs will stay the same. Just look at how much it would have taken to train something like GPT-3 five years ago.