News for hackers. Stuff that matters.

A comment on Differentiable Programming – A Simple Introduction. Parent comment

Re:Differentiable Programming – A Simple Introduction by PartiallyTyped on Wednesday April 13, 2022 @01:27PM (#31017466)
Well, quadratic convergence usually requires the Hessian, or an approximation of it, and that's difficult to get in deep learning due to memory constrains, and difficulty computing second order derivatives.

Computing the derivatives is not very difficult with e.g. Jax, but ... you get back to the memory issue. The Hessian is a square matrix, so in Deep Learning, if we have a million of parameters, then the Hessian is a 1 trillion square matrix...

[ Parent | on HN ]
Re:Differentiable Programming – A Simple Introduction by tome on Wednesday April 13, 2022 @01:43PM (#31017711)
Not only does it have 1 trillion elements, you also have to invert it!
[ Parent | on HN ]
Re:Differentiable Programming – A Simple Introduction by PartiallyTyped on Wednesday April 13, 2022 @02:19PM (#31018254)
Indeed! BFGS (and derivatives) approximate the inverse but they have other issues that make them prohibitively expensive.
[ Parent | on HN ]
Re:Differentiable Programming – A Simple Introduction by SleekEagle on Wednesday April 13, 2022 @02:00PM (#31017964)
https://c.tenor.com/enoxmmTG1wEAAAAC/heart-attack-in-pain.gi...
[ Parent | on HN ]
Re:Differentiable Programming – A Simple Introduction by ssivark on Thursday April 14, 2022 @12:17AM (#31023353)
To add, one could think of schemes like "momentum" and cousins as attempts to estimate something in the spirit of the inverse Hessian using various hacks/heuristics.
[ Parent | on HN ]