News for hackers. Stuff that matters.
Re:Techniques for Training Large Neural Networks by pchiusano on Thursday June 09, 2022 @01:59PM (#31684575)
You might like these posts on gradient free optimization:

http://pchiusano.github.io/2019-07-06/learning-without-a-gra...

http://pchiusano.github.io/2020-10-12/learning-without-a-gra...

I think you’re right that the area is underexplored.

Re:Techniques for Training Large Neural Networks by eismcc on Thursday June 09, 2022 @03:19PM (#31685591)
Has anyone tried using another NN as the “gradient” function? With the hope of making it less “blind?”
Re:Techniques for Training Large Neural Networks by rolisz on Thursday June 09, 2022 @04:13PM (#31686184)
DeepMind has a thing called synthetic gradients: https://www.deepmind.com/blog/decoupled-neural-interfaces-us....