If the language defines a way for you to ask the program for the computation graph in order to differentiate -- couldn't you add meta programming functions on top of that computation graph?

Seems to me like there's at least an argument that it should be possible grow diff programming into a general purpose meta programming system rather than use general purpose meta programming as implementation mechanism for differentiable programming ...