Moore's Law, AI, and the pace of progress

Re:Moore's Law, AI, and the pace of progress by visarga on Sunday December 12, 2021 @02:53AM (#29527883)

Didn't say anything about algorithms improving faster than hardware. With better initialization and regularization, more efficient layers discovered by AutoML, sparsity, lower precision, or by directly combining models trained separately on diverse tasks instead of training sequentially we could have better models at the same hardware cost.

Even by improving the way people collaborate on model training we could make a huge leap. Instead of retraining from scratch we could reuse and update existing models. [1]

[1] A Call to Build Models Like We Build Open-Source Software https://colinraffel.com/blog/a-call-to-build-models-like-we-...

[ Parent | on HN ]

Re:Moore's Law, AI, and the pace of progress by sgt101 on Sunday December 12, 2021 @04:55AM (#29528335)

I keep reading about how AutoML will make things more efficient, but in my experience it's been hideously expensive in both time and cost (and I am therefore guessing energy), and had yielded marginal improvements at best. Often I have encountered AutoML in the form of an overfit model with test data leaks that has underperformed in prod having overperformed in test.

So - I don't see how AutoML is going to discover more efficient layers without expending loads of energy, can you describe how it will contribute to lowering the whole energy budget?

On the models as oss thing - I think everyone likes this idea, but models are not the same as code andoss doesn't always produce good code cf. desktop linux.

[ Parent | on HN ]