That much I understood myself.
> His description of reverse mode AD is more complicated
And that's where I got lost in the weeds.
I'm sure if I could be bothered, I could sit down for a few weeks or months and gain a working understanding of Category Theory.
My point is not whether I understand or don't understand CT, but... what's the point of spending effort in understanding it?
The CT of forward and reverse mode Automatic Differentiation is cute, but it doesn't seem to have had much practical application in precisely the area where CT ought to be useful.
I mean... both reverse and forward mode AD existed before it was labelled using CT terminology. What did CT add to the SOTA for AD?
So the thing it added is a compact and simple (and therefore pedagogically useful) description of AD for people with the target background.
I was the one who linked that. I wasn't familiar with AD before his talk, but I find it perfectly understandable (modulo Haskell's sometimes foreign-to-me syntax). I think the relevant background here is that I was a math major and had a course in abstract linear algebra (i.e. general vector spaces over fields) and two semesters of abstract algebra (groups, rings, fields) plus dipped my toes in some CT on my own.
He basically notes that D had a slightly nicer chain rule if you write D'f(a)= (f(a), Df(a)) and use D' instead. Then his definition of AD is just the chain rule and the product rule, plus the fact that Df = f for all linear f, applied recursively. So forward mode AD is just writing down facts you learned in calc 1 as a recursive definition (with, I guess, some base cases like sin/cos hard coded).
His description of reverse mode AD is more complicated, but IMO still accessible to someone who's seen dual spaces, product/coproduct vector spaces and maybe enough CT to understand that the map from a vector space to its dual (and matrices to their adjoint) is a contravariant functor.
The basic observation is that he can write the chain rule and product rule as a compositions of linear maps with product/coproduct spaces (which are the same thing for vector spaces). Then he says a biproduct category is one that has (equal) product/coproduct, and notes that the dual map preserves that structure.
Since he wrote D' recursively in terms of linear maps, and dualizing/taking the adjoint still gives him the structure he needs to do that, he's able to write reverse mode AD as the adjoint representation of D'. He just needs to write the adjoints of the pieces he used, and again computer recursively.
I know at least at my school, the engineering program I was in had a first semester graduate class that dealt some with dual representations (using "test functions" etc. when solving differential equations), so I think it could be distilled down to something appropriate for masters level engineering while retaining the character of what he presented. Like I said, the Haskell syntax was probably the most obscure thing for me.
It's all a matter of perspective and background, but IMO it's sort of like how if you're familiar with linear algebra (including function spaces), then the Fourier transform is just writing a function as a sum of projections onto an orthonormal basis. No need to remember the formulas for it; they're obvious and have simple geometric meaning.