News for hackers. Stuff that matters.
Re:Symbolic Discovery of Optimization Algorithms by cs702 on Friday February 17, 2023 @10:45PM (#34843665)
This looks great!

The authors are reputable and claim to have discovered a simple lr optimizer than trains models up to 5x faster and converges to better local optima than Adam/AdamW.

As usual, YMMV.

You can find a PyTorch implementation here: https://github.com/lucidrains/lion-pytorch

Preliminary tests seem to confirm the paper's claims.

There are no comments on this comment yet.