Re:Symbolic Discovery of Optimization Algorithms
by cs702 on Friday February 17, 2023 @10:45PM
(#34843665)
This looks great!
The authors are reputable and claim to have discovered a simple lr optimizer than trains models up to 5x faster and converges to better local optima than Adam/AdamW.
The authors are reputable and claim to have discovered a simple lr optimizer than trains models up to 5x faster and converges to better local optima than Adam/AdamW.
As usual, YMMV.
You can find a PyTorch implementation here: https://github.com/lucidrains/lion-pytorch
Preliminary tests seem to confirm the paper's claims.