Re:Techniques for Training Large Neural Networks
by pchiusano on Thursday June 09, 2022 @01:59PM
(#31684575)
You might like these posts on gradient free optimization:
http://pchiusano.github.io/2019-07-06/learning-without-a-gra...
http://pchiusano.github.io/2020-10-12/learning-without-a-gra...
I think you’re right that the area is underexplored.
Re:Techniques for Training Large Neural Networks
by eismcc on Thursday June 09, 2022 @03:19PM
(#31685591)
Has anyone tried using another NN as the “gradient” function? With the hope of making it less “blind?”
Re:Techniques for Training Large Neural Networks
by rolisz on Thursday June 09, 2022 @04:13PM
(#31686184)
DeepMind has a thing called synthetic gradients: https://www.deepmind.com/blog/decoupled-neural-interfaces-us....
Re:Techniques for Training Large Neural Networks
by erwincoumans on Thursday June 09, 2022 @04:25PM
(#31686331)
Black box optimization such as CMA-ES or Augmented Random Search (ars) are used by researchers, for example:
CMA-ES: https://blog.otoro.net/2017/10/29/visual-evolution-strategie...
ARS: https://arxiv.org/abs/1803.07055
Those optimizers are efficient up to certain neural network sizes.
https://openai.com/blog/evolution-strategies/
Put engineering effort into creating fast, GPU powered (tensorflow/pytorch) algorithms for neuroevolution of the weights of a neural network. I'm still not convinced that properly leveraged gradient free/Feed-forward only optimizers have been tried yet by researchers - namely because they never actually wrote fast gradient free optimizers!