News for hackers. Stuff that matters.
Re:Symbolic Discovery of Optimization Algorithms by im3w1l on Saturday February 18, 2023 @09:58AM (#34847344)
Optimization algorithm they arrived at was (quoting from the paper)

  def train(weight, gradient, momentum, lr):
    update = interp(gradient, momentum, β1)
    update = sign(update)
    momentum = interp(gradient, momentum, β2)
    weight_decay = weight * λ
    update = update + weight_decay
    update = update * lr
    return update, momentum
The key thing they do, sending the update through sign, means all weight updates turn into either 1 or -1.
Re:Symbolic Discovery of Optimization Algorithms by nestorD on Saturday February 18, 2023 @10:34AM (#34847620)
1 or -1 times the learning rate.