Being able to calculate the derivative is very useful (the backprop rule, etc.). However it isn't actually necessary here, since you can approximate the slope by just sampling points nearly. Strict differentiability isn't the crucial property here, but it does make things faster.

What is, I think, is that the error function is continuous and there is a monotonic path of improvement from most of the space to good solutions in it. Then you can just descend in the error function until you reach it.

We don't really understand why that happens in deep learning, that is, why the error functions of deep neural networks tend to not have complex shapes with local minima that you can get stuck in. This is currently being investigated.

This also isn't a problem limited to deep learning. In evolution, the same is true - we are limited to small local improvements, but over large amounts of time nature arrives at amazingly efficient solutions. That suggests that the fitness function space over genomes is also, to some extent, continuous and allows monotonic improvement. Again, we don't really know why, yet.

Finite difference approximations to the derivative are not practical at all when doing deep learning. Derivative using backpropagations can be calculated at the same speed as the forward pass - while derivative approximated using finite differences require 2 passes PER parameter, and run into numerous numerical issues with accuracy.

It's an absolute non-starter.

You don't actually need to calculate the derivative accurately. It's enough to sample several points near the current one and continue the search to the best of those.

The point here is that we are descending in a continuous and monotonic loss function. You don't need an accurate derivative or even an estimate of the derivative to do so, although it definitely helps a great deal.

Except that you are talking about the derivative with respect to each weight. In a deep neural network, you'd need several sample points for each weight in order to compute dC/dw. That is a LOT of work for a single step. The only way to do this in a timely fashion is via backprop, which requires a derivative.

is there a term or subject area where one could learn more about the investigations underway on error functions that tend to not have complex shapes with local minima to get stuck in?

It's called convex analysis, here's a course on the topic: https://lagunita.stanford.edu/courses/Engineering/CVX101/Win...

Just to be clear here, your meaning is that the reward function should be differentiable so there is a slope to climb to a maxima (local or gobal).

- edit used the wrong word, used derivative meant differentiable.

He's talking about the operations needing to be differentiable. So like multiplication, addition etc are differentiable. tanh, sigmoid functions are differentiable. You need this because back propagation is an application of the chain rule, so each constituent operation needs to be differentiable.

A lot of interesting advances have come from making operations that seems like they aren't differentiable, into differentiable operations that can be trained by backpropagation. See LSTMs and neural turing machines