Finite difference approximations to the derivative are not practical at all when doing deep learning. Derivative using backpropagations can be calculated at the same speed as the forward pass - while derivative approximated using finite differences require 2 passes PER parameter, and run into numerous numerical issues with accuracy.
It's an absolute non-starter.
You don't actually need to calculate the derivative accurately. It's enough to sample several points near the current one and continue the search to the best of those.
The point here is that we are descending in a continuous and monotonic loss function. You don't need an accurate derivative or even an estimate of the derivative to do so, although it definitely helps a great deal.
Except that you are talking about the derivative with respect to each weight. In a deep neural network, you'd need several sample points for each weight in order to compute dC/dw. That is a LOT of work for a single step. The only way to do this in a timely fashion is via backprop, which requires a derivative.
Re:What’s Next for Artificial Intelligence
by TumTumTree on Monday June 20, 2016 @02:19PM
(#11939987)
is there a term or subject area where one could learn more about the investigations underway on error functions that tend to not have complex shapes with local minima to get stuck in?
Re:What’s Next for Artificial Intelligence
by kirill5pol on Tuesday June 21, 2016 @10:53AM
(#11946034)
It's called convex analysis, here's a course on the topic: https://lagunita.stanford.edu/courses/Engineering/CVX101/Win...
What is, I think, is that the error function is continuous and there is a monotonic path of improvement from most of the space to good solutions in it. Then you can just descend in the error function until you reach it.
We don't really understand why that happens in deep learning, that is, why the error functions of deep neural networks tend to not have complex shapes with local minima that you can get stuck in. This is currently being investigated.
This also isn't a problem limited to deep learning. In evolution, the same is true - we are limited to small local improvements, but over large amounts of time nature arrives at amazingly efficient solutions. That suggests that the fitness function space over genomes is also, to some extent, continuous and allows monotonic improvement. Again, we don't really know why, yet.