News for hackers. Stuff that matters.
Re:From automatic differentiation to message passing [video] by taliesinb on Thursday August 15, 2019 @11:49AM (#20706255)
This is a perspective I’ve been taking recently as I’ve been investigating next-gen DL frameworks like Julia’s Zygote and Swift for Twnsorflow. So it’s very nice to see this articulated so well here by someone who has been thinking deeply about it for so long.

I’ll add some things that relate to this perspective: there is a follow up paper to the DL via Hessian-free optimization paper by James Martens that develops a variant of AD which calculates a special curvature quantity which is useful for efficient second order optimization. This emphasizes that AD is just one of a family of program transformations, most of which are probably waiting to be discovered. For example, would it be useful to have an AD-like pass that calculated trusted regions for gradient updates? And: why do we not do more iteration during DL in which we interleave forward and backward for a single layer only? (rhetorical question, there are many reasons but asking this question forces you to think deeply about a lot of things)

Also, if you generalize VAEs to be arbitrary differentiable programs you immediately run into versions of message passing. For example, vector quantized VAEs are a precisely a soft version of the EM step in k-means. So how would we develop more advanced variational programs in which this process is better managed? There may be very different forms for the inference and generation phases that are not related by a program transformation, or ARE related but only in a high-level way best captured by functional programming primitives.

Any other people nerding out about this stuff please say hi!

Re:From automatic differentiation to message passing [video] by proofofconcept on Thursday August 15, 2019 @01:43PM (#20707612)
>there is a follow up paper to the DL via Hessian-free optimization paper by James Martens that develops a variant of AD which calculates a special curvature quantity which is useful for efficient second order optimization

Deep Learning via Hessian-free Optimization: http://www.cs.toronto.edu/~jmartens/docs/Deep_HessianFree.pd...

Optimizing Neural Networks with Kronecker-factored Approximate Curvature: http://arxiv.org/abs/1503.05671

James Martens' list of publications with links to sample code for the above two papers, slides/condensed conference versions, etc: http://www.cs.toronto.edu/~jmartens/research.html

Pretty neat stuff

Re:From automatic differentiation to message passing [video] by taliesinb on Thursday August 15, 2019 @07:13PM (#20710598)
Haha thanks for finding links! Was on a bus on my phone, so didn't have the patience...
Re:From automatic differentiation to message passing [video] by taliesinb on Thursday August 15, 2019 @07:16PM (#20710623)
> For example, would it be useful to have an AD-like pass that calculated trusted regions for gradient updates?

To answer my own question, yes: http://papers.nips.cc/paper/7112-scalable-trust-region-metho...