News for hackers. Stuff that matters.
Re:Differentiable Programming Manifesto by kragen on Wednesday November 13, 2019 @01:23AM (#21521588)
This looks like it's going to be a really big deal unless they totally fuck it up. Autodiff greatly simplifies all kinds of continuous mathematical optimization algorithms, and using mathematical optimization means you only have to write code that can recognize things that look similar to a solution instead of code that can find the solution — in a sense, mathematical optimization derives your code from tests.

Autodiff is useful for other kinds of things too, not just mathematical optimization, as they mention in the manifesto. This weekend I wrote a real-time SDF raymarcher in a page of Lua (https://gitlab.com/kragen/bubbleos/blob/master/yeso/sdf.png https://gitlab.com/kragen/bubbleos/blob/master/yeso/sdf.lua), but the surface shading is pretty lame because without sampling the SDF multiple times it doesn't yet have any way to calculate surface normals. Guess what a surface normal is? It's the gradient of the SDF, and reverse-mode autodiff can sample the SDF and its gradient in only double the time of just sampling the SDF.

Pretty much anything you do in continuous domains can get simpler and/or better-performing with autodiff. This is why I've written so much about it in Dercuano.

It's worth pointing out that you don't need to redesign your language to get first-class differentiable programming. Source-to-source transformation has been used to do autodiff in FORTRAN for at least ten years, including on pretty massive applications. (This is mentioned in the manifesto.)

Re:Differentiable Programming Manifesto by seanmcdirmid on Wednesday November 13, 2019 @09:29AM (#21524435)
Conal Elliott (the guy who co-invented FRP) did a nice project on shading computational surfaces by extracting derivatives from code more than a decade ago: http://conal.net/Vertigo/

He is now very active in the differentiable programming community, but he got started on the problem in the graphics field.

Re:Differentiable Programming Manifesto by noblethrasher on Wednesday November 13, 2019 @10:34AM (#21525236)
He also gave nice presentation on AD: https://www.youtube.com/watch?v=ne99laPUxN4 (surprised that you didn't already mention it, considering the venue ;-))
Re:Differentiable Programming Manifesto by kragen on Wednesday November 13, 2019 @11:50AM (#21526197)
The manifesto links one of his papers, "The simple essence", which I've started but haven't finished.
Re:Differentiable Programming Manifesto by seanmcdirmid on Wednesday November 13, 2019 @12:42PM (#21527014)
He has been working on that for a long time. Though much of his earlier work is symbolic differentiation, I’m not very familiar with the more recent AD stuff.
Re:Differentiable Programming Manifesto by thechao on Thursday November 14, 2019 @08:20AM (#21535129)
Let me mostly summarize Conal’s “simple” reverse AD: he drops the tape into the stack. This has two impacts: the stack is “about twice as long”; and, you “may (will?) lose tail recursion”.

It is still a really good paper—and no doubt a lot of the subtlety was list to me.

Re:Differentiable Programming Manifesto by sitkack on Wednesday November 13, 2019 @12:50PM (#21527106)
AD also has a nice synergy with DNN, the whole formalization allows for moving the runtime into a DNN and thus onto a GPU.

I have been looking for a technique for representing algorithms that are amenable to conversion to anytime [1] I believe AD could be that technique.

[1] https://en.wikipedia.org/wiki/Anytime_algorithm

Re:Differentiable Programming Manifesto by kragen on Wednesday November 13, 2019 @03:49PM (#21529379)
Well, successive-approximation mathematical optimization algorithms are sort of inherently anytime, in a way: you can stop after any integer number of iterations and use the approximate solution. Some of them fail to be anytime in practice, like SOR (“successive over-relaxation”) where you usually quit after like six iterations but each iteration takes a long time. And autodiff makes it a lot easier to apply mathematical optimization, in particular variants of gradient descent, one popular use of which is training deep ANNs.

Is that what you meant, or are you talking about something else?

Re:Differentiable Programming Manifesto by sitkack on Wednesday November 13, 2019 @04:58PM (#21530069)
This is it.

I think AD can be used to train a DNN, the DNN in turn can be used to power an anytime algorithm such that you can have systems that can not only respond on different time scales, they are also robust in the face of sensor noise, adversarial sensors or broken actuators.