Re:It takes a lot of energy for machines to learn
by joe_the_user on Wednesday December 30, 2020 @08:05PM
(#25588359)
ML hardware is no doubt progressing. Whether we can expect a rapid rate is a different matter. The various Moore's Laws have been breaking down (chip speed stopped doubling a while back). GPU are the sort of chip that could most benefit from just transistors doubling (last Mooore's law standing) but even this Moore's Law is breaking down.
ML model size has had it's own Moore's Law, with standard model growing exponentially in size [1]. And this implies model are going butt up against the limits even more than they have already. Whether the "bitter lesson"[2] of ML is true inherently is up for a question. That current researchers have accepted it seems given.
ML model size has had it's own Moore's Law, with standard model growing exponentially in size [1]. And this implies model are going butt up against the limits even more than they have already. Whether the "bitter lesson"[2] of ML is true inherently is up for a question. That current researchers have accepted it seems given.
[1] https://openai.com/blog/ai-and-compute/
[2] http://www.incompleteideas.net/IncIdeas/BitterLesson.html