News for hackers. Stuff that matters.
Re:Mathematics of Deep Learning [pdf] by msmith10101 on Saturday December 16, 2017 @07:32PM (#15943163)
Newtons method and linear algebra. High school math, basically.
Re:Mathematics of Deep Learning [pdf] by aisofteng on Sunday December 17, 2017 @06:04PM (#15948181)
>For instance, local entropy, the loss that Entropy-SGD minimizes, is the solution of the Hamilton- Jacobi-Bellman PDE and can therefore be written as a stochastic optimal control problem, which penalizes greedy gradient descent. This direction further leads to connections between variants of SGD with good empirical performance and standard methods in convex optimization such as inf- convolutions and proximal methods.

You clearly didn't read the article. What are you commenting on, then? It seems to be about your own understanding, since it's certainly not about the article.

Re:Mathematics of Deep Learning [pdf] by santaclaus on Saturday December 16, 2017 @08:01PM (#15943279)
> Newtons method

Where are you seeing Newton's method? I didn't think second order information was available for typical systems in statistical machine learning.

Re:Mathematics of Deep Learning [pdf] by make3 on Saturday December 16, 2017 @10:37PM (#15943932)
my understanding is that the issue is that the full Hessian of the loss is too expensive to compute at each step for the relative size of the increase in learning speed
Re:Mathematics of Deep Learning [pdf] by _0ffh on Sunday December 17, 2017 @06:04AM (#15945251)
Yeah I think that's why quasi-Newton methods like BFGS have been developed.