News for hackers. Stuff that matters.
Re:Automatic Differentiation in Machine Learning: A Survey [... by kxyvr on Monday November 19, 2018 @10:19PM (#18491599)
I skimmed the article and I enjoyed it. Recently, I was looking for a good reference to cite that discussed the link between automatic differentiation and back propagation. From those I know if the field, they knew about the link for a very, very long time, but I had a hard time finding a paper to cite. The best I found was an article titled, "Backwards Differentiation in AD and Neural Nets: Past Links and New Opportunities" by Paul Werbos, which can be found in the book "Automatic Differentiation: Applications, Theory, and Implementations" published in 2006 by Springer. That article is preceded by one written by Louis Rall titled, "Perspectives on Automatic Differentiation: Past, Present, and Future?", which also provides some good history of AD. Anyway, if someone has a better paper to cite linking back propagation and AD, I'd be interested to hear.
Re:Automatic Differentiation in Machine Learning: A Survey [... by whatshisface on Monday November 19, 2018 @10:25PM (#18491622)
I thought the article itself did a good job of explaining it. Backpropagation is just a special case of one way to do AD, apparently. AD prescribes a method of taking derivatives, and if you use it on a neural network, there you go, backpropagation.