Re:How Fast Do Algorithms Improve? [pdf]
by tedsanders on Thursday September 23, 2021 @05:45PM
(#28635936)
Progress in computation can be decomposed into three factors:
Perf = (perf / op) * (op / $) * $
(1) Perf / op is essentially software progress, the subject of this article. (And for AI/ML, you can further decompose it into algorithm progress and training data progress.)
(2) Op / $ is essentially Moore's law, which has been slowing down and can be further decomposed into: Ops / $ = (ops / transistor) * (transistor / area) * (area / $)
(3) $ is willingness to spend money. This has probably been the biggest source of progress for AI projects in the past 10 years, with Deep Mind and Open AI and Google Brain and Silicon Valley ad tech spending billions to build models more complex and capable than any that have come before. http://www.rossgritz.com/trends-in-deepmind-operating-costs/...
Perf = (perf / op) * (op / $) * $
(1) Perf / op is essentially software progress, the subject of this article. (And for AI/ML, you can further decompose it into algorithm progress and training data progress.)
(2) Op / $ is essentially Moore's law, which has been slowing down and can be further decomposed into: Ops / $ = (ops / transistor) * (transistor / area) * (area / $)
(3) $ is willingness to spend money. This has probably been the biggest source of progress for AI projects in the past 10 years, with Deep Mind and Open AI and Google Brain and Silicon Valley ad tech spending billions to build models more complex and capable than any that have come before. http://www.rossgritz.com/trends-in-deepmind-operating-costs/...