Re:Automatic Differentiation (2009)
by abecedarius on Saturday January 02, 2016 @02:19AM
(#10825426)
Yes, that's one way to look at it. However, here's a Python symbolic differentiator I just coded now to make a more direct comparison, since Norvig's Lisp is very different from my Python in lots of distracting ways: https://github.com/darius/sketchbook/blob/master/misc/symbdi...
This symbolic differentiator could be made to not blow up, by memoizing, i.e. decorating with https://github.com/darius/sketchbook/blob/master/regex/memo.... . It'd still have the difference that it constructs an explicit formula in x, as a data structure; the autodiff code instead takes a particular x as input and calculates a value for just that x, where the value includes both an 'actual value' and a 'derivative' component. (Your library and compiler could get fancy and inline that computation at compile time back into a pair of formulas, and then yes, they ought to be exactly equivalent.)
Re:Automatic Differentiation (2009)
by throwaway39210 on Saturday January 02, 2016 @05:25AM
(#10825768)
Thanks, these posts are the only thing I've read that have made sense to me thus far (probably because they mostly confirm my suspicion, that there's not really a strong distinction between 'symbolic' and 'automatic' differentiation, but there are perceived differences based on whether the computation graph is compiled and optimized, and whether the graph is pretty-printed at some point...). Your Python symbolic differentiator is exactly a mini-Theano (or a mini-any deep learning framework). Each `Function` instance (or `Op` in Theano) defines its output and gradient with respect to its input (given gradient wrt output).
That's essentially what autodiff is. It works on the DAG, whereas "symbolic" diff insists that the DAG is a tree (so reused components have to be duplicated, which leads to blowup).