Re:From scratch: reverse-mode automatic differentiation (in ...
by a1369209993 on Monday June 15, 2020 @03:00AM
(#23524567)
Technically there is:
D f = f' <=> S f' = f # inverse
# chain rule
D (f . g) x = D f (g x) * D g x
# divide by D g x
D (f . g) x / D g x = (D f) (g x)
# S vs D inverse
D (S f' . g) x / D g x = (f' . g) x
# commute = and integrate both sides
S (f' . g) x = S \x( D (S f' . g) x / D g x ) x
# rename and drop argument
S (f . g) = S \x( D (S f . g) x / D g x )
The problem is that the RHS is, in the general case, not any less complex (and often much more) than the LHS.
Edit: Note that (S f . g) is integrating only f, then composing with g, which is what you want; it's the S \x(...) that causes problems.
Edit: Note that (S f . g) is integrating only f, then composing with g, which is what you want; it's the S \x(...) that causes problems.