News for hackers. Stuff that matters.

A comment on A Bayesian Perspective on Generalization and Stochastic Gradient Descent.

Re:A Bayesian Perspective on Generalization and Stochastic G... by evrydayhustling on Saturday December 08, 2018 @03:56PM (#18637484)
This result is an amazing twofer. Not only do they give a formal probabilistic perspective on how deep nets fall into local minima, the result proposed a way to reduce two "voodoo parameters" - learning rate and batch size - to one, by showing how to pick the other in a way that is optimal for expected generalization.
[ Parent | on HN ]
Re:A Bayesian Perspective on Generalization and Stochastic G... by mlthoughts2018 on Saturday December 08, 2018 @05:03PM (#18637796)
I suspect actually there is some connection between optimal batch size and the proofs of this paper regarding how deep NN will provably converge to zero training loss with gradient descent (not stochastic gradient descent).

https://arxiv.org/abs/1811.03804

[ Parent | on HN ]
Re:A Bayesian Perspective on Generalization and Stochastic G... by samgd on Sunday December 09, 2018 @05:07AM (#18640142)
You should check out our blog post related to finding the optimal learning rate, batch size, and momentum setting:

https://www.myrtle.ai/2018/09/24/how-to-train-your-resnet-5/

[ Parent | on HN ]