There is a whole sub field of mathematics / computer science called "algorithmic differentiation", also known as "automatic differentiation".

The goal is to take an existing computer program and transform it into another computer program that calculates the derivatice just as efficiently (up to a constant factor of ~2 to 5, depending on how you measure), and with about the same numeric stability (by usual measures).

And this is just the start. Classic differentiation is also called "forward" mode. Understanding the "reverse mode", and why you want this for gradients, is even more mind-bending.

There is a whole world to discover here. The book "Evaluating Derivatives" is a good starting point for anyone interested.

http://epubs.siam.org/doi/book/10.1137/1.9780898717761

There are also many of beautiful papers on that topic.

Yes, there are combined forward-reverse-mode approaches as well as "cross country" approaches. That's in part of what I menat with a "whole world to discover".

> Not only that, but most of the difficulty is in getting these tools to work on arbitrary codebases, making it more a software engineering problem rather than a maths problem

Depending on your viewpoint, it is neither SE nor math, but a PL (programming languages) issue, meaning that viewing the program e.g. as register machine (classic assembly) yields to different approaches than viewing it as lambda calculus expression (which provided great results, but it is hard to transport the results back to languages like Fortran or C++).

[0] https://en.m.wikipedia.org/wiki/Dual_number