News for hackers. Stuff that matters.
Re:Symbolic Discovery of Optimization Algorithms by yobbo on Saturday February 18, 2023 @11:48AM (#34848321)
Yes. I omitted time-indices, and moved the momentum-update up to before the parameter-update. In their pseudo-code θₜ updated with mₜ₋₁ whereas above, it would be with mₜ.

I suspect it is equivalent up to slightly different α and β?

Re:Symbolic Discovery of Optimization Algorithms by mtcrawshaw on Saturday February 18, 2023 @12:31PM (#34848868)
Moving the momentum update before the parameter update changes the algorithm from LION to SignSGD. The only novel part of their algorithm is the fact that the momentum changes after the update is computed. They are not equivalent even if you can choose coefficients however you like. But as I said, if I had to guess I would say that this does not cause a fundamental change in the behavior of the optimization.
Re:Symbolic Discovery of Optimization Algorithms by yobbo on Sunday February 19, 2023 @04:34AM (#34855932)
If we let βₛ be as above (SignSGD) and βₗ be as in LION, then it looks to me that βₛ=βₗ/α should make c from both equal.