Re:Composability in Julia: Implementing Deep Equilibrium Mod...
by ChrisRackauckas on Thursday October 21, 2021 @02:37PM
(#28949077)
Yeah, though with the method described in that paper you do have to be very careful since it has exponential error growth with the Lipchitz constants of the ODE. See https://aip.scitation.org/doi/10.1063/5.0060697 for details. But that is generally the case in numerical analysis: there's always a simple way to do things, and then there's the way that prevents error growths. Both have different pros and cons.
The basic idea is inspired by the “adjoint method” for ODE solving (so you don’t have to hold in memory all the intermediate layer outputs — which is otherwise necessary to compute the backpropagated gradient signal).