You might be interested in David MacKay's old conjugate gradient implementation in C that also only uses derivatives: https://www.inference.org.uk/mackay/c/macopt.html

(It was written in another time, don't expect it to be nice. My Octave/Matlab wrappers on that page have almost certainly bit-rotted, don't expect them to work.)