News for hackers. Stuff that matters.
Re:DeepMind achieves SOTA image recognition with 8.7x faster... by longtom on Sunday February 14, 2021 @12:35PM (#26134260)
tl;dr: Don't use batch norm for preventing exploding gradients but adaptive gradient thresholds.

For this they compute the Frobenius norm (square root of the sum of squares) of the weight layer and its gradient and take the ratio of these as clipping threshold.

That saves the meta search for the optimal threshold but also is better than a fixed threshold could ever be.

Very simple idea.

Re:DeepMind achieves SOTA image recognition with 8.7x faster... by sillysaurusx on Sunday February 14, 2021 @12:39PM (#26134289)
Thanks for macroexpanding frobnorm.

I'm skeptical that these hand-coded thresholds can ever match what a model can learn automatically. But it's hard to argue with results.

Re:DeepMind achieves SOTA image recognition with 8.7x faster... by highfrequency on Sunday February 14, 2021 @12:44PM (#26134340)
Is this the first time that the gradient clip threshold has been chosen relative to the size of the weight matrix?
Re:DeepMind achieves SOTA image recognition with 8.7x faster... by longtom on Sunday February 14, 2021 @04:08PM (#26136278)
Lol, why is my comment down here with 7 upvotes.