Re:Differentiation for Hackers
by tobmlt on Saturday June 22, 2019 @01:43PM
(#20250772)
The differentiation of a fractional power (fraction less than 1.) involves a negative fractional power. 0 raised to some negative power involves dividing by zero. This gives you some kind of NAN in, e.g., vanilla languages such as Python. The "issue with automatic differentiation" is really an issue with dividing by zero. Graph x(1./3.). It's slope is vertical at zero.
[ Parent
| on HN ]
I don't think this is right. An autodiff package will likely report +inf for the derivative of cuberoot at 0.
The problem is really that there are a lot of ways the function can have infinite slope. The sqrt function also has infinite slope at 0.
Dividing by zero isn't where the NaN comes from.
My mistake then. I did not intend to try and be so precise. I was thinking loosely. I have a very simple autodiff tool hand rolled (in Python) here to play with. I raise 0 to a negative exponent in the gradient of the cube root at 0. I do not get a NAN. Strictly speaking I get this:
"ZeroDivisionError: 0.0 cannot be raised to a negative power"
Is that satisfying? I do not claim anything more specific here. Sorry to be imprecise.