To follow up on this, a lot of the overhead from reverse-mode AD can be avoided by flattening out the expression graphs and compiling specialized functions (at runtime). A number of the JuliaDiff packages support this approach. This is not a new idea, but Julia's hooks to LLVM make this surprisingly easy to do.