Re:Differentiation for Hackers
by FooBarBizBazz on Saturday June 22, 2019 @05:56PM
(#20251919)
Actually I'm wrong about L'Hospital's Rule in this problem; it solves nothing here. I would insert an "[EDIT ...]" into my original post to avoid misleading people, but it's too late to edit.
It doesn't matter how many times you differentiate x^(1/3); you can not evaluate the result at zero. Which is funny, because you'd think you could differentiate x^3 and use the inverse function theorem. But there's that caveat about it not working when the derivative is zero.
I am now revisiting high school calculus and doubting my own sanity. Will post updates if/when I get my head right.
Re:Differentiation for Hackers
by FooBarBizBazz on Saturday June 22, 2019 @03:57PM
(#20251428)
Here's the (pedantically-worked) example with reverse-mode:
Program:
y = g(x) = x^3
z = f(y) = cuberoot(y)
Reverse-mode autodiff:
// Forward pass
y = g(0) = 0^3 = 0 g'(0) = 3 x^2 = 0
z = f(0) = cuberoot(0) = 0 f'(0) = (1/3) / cuberoot(y^2) = (1/3)/0 = Inf
// Reverse pass
dz/dy = f'(0) = Inf
dz/dx = dz/dy * g'(0) = Inf * 0 = NaN
I'd realized this was an issue with forward-mode autodiff, but hadn't thought about reverse-mode to realize it comes up there too.
In general, NaN is a placeholder for "all the derivatives you didn't know you needed in advance". I think the Dual Numbers shed light on this (see "Division": [https://en.wikipedia.org/wiki/Dual_number#Division]).
One could imagine lazily computing higher derivatives as needed for L'Hospital's Rule in either Forward or Reverse mode, however. For Forward Mode, I guess you'd replace your eager "Dual Number" pairs with lazy lists?
For reverse mode it seems you could do extra "passes" on the tape as needed... People who do this stuff for a living must know(?), but a quick Google doesn't turn up an answer for me...