News for hackers. Stuff that matters.
Re:How AI Training Scales by taliesinb on Friday December 14, 2018 @02:40PM (#18683898)
This is an exciting work.

Gradient noise scale would be interesting to compute individually for each weight/bias. I'm curious how it scales as a function of layer depth, for example -- it could well be that different layers require very different treatment in terms of learning rate. Supporting this is that gradient magnitudes are typically very different for different layers, although they evolve similarly in time. Here's something I just whipped up in Mathematica:

https://imgur.com/a/HhCnlDL

Re:How AI Training Scales by samsamoa on Friday December 14, 2018 @03:25PM (#18684358)
Author here— Yes, this is a great question! I think there's a lot more interesting work to be done here. It would be great if we could understand e.g. why layer-wise learning rates help with large-batch training of ImageNet (https://arxiv.org/abs/1708.03888), and maybe the per-component noise scale has something to do with it.