I assume backprop is still most efficient for the neural net itself, rather than one of these algorithms? I am doing a course and I am aware that there are other algorithms than gradient descent but haven’t seen the details yet on that.

I know things like cross entropy loss are designed to be easy to differentiate and probably more efficient to do that than something that approximates it.

Yeah, backprop gives efficient gradient calcs to train at a given net architecture. To find the best architecture though people try various other methods, eg random search or Gaussian processes, which don’t evaluate derivatives wrt architecture Params.

You might be interested in David MacKay's old conjugate gradient implementation in C that also only uses derivatives: https://www.inference.org.uk/mackay/c/macopt.html

(It was written in another time, don't expect it to be nice. My Octave/Matlab wrappers on that page have almost certainly bit-rotted, don't expect them to work.)

Thank you! I’ll definitely check this out.

> It finds roots when you can only measure the derivative.

How is that even possible? The function f(x)=x-7 has a single root at x=7. Its derivative is f'(x)=1. How do you recover 7 from 1 ?

You need one initial value to start from