News for hackers. Stuff that matters.
Re:DeepMind achieves SOTA image recognition with 8.7x faster... by belgian_guy on Sunday February 14, 2021 @02:46PM (#26135540)
Yes they do (arxiv is a place for scientific papers not press releases). I've only skimmed it, but the paper introduce an adaptive way to clip gradients. Meaning that if the ratio of the gradient norm to weight norm surpasses a certain threshold, they clip it. This stabilizes learning and seems to avoid the need for batch normalization. Seems quite promising imo and something that could stick (I'm quite happy if we could finally do away with batchnorm).
Re:DeepMind achieves SOTA image recognition with 8.7x faster... by eggie5 on Sunday February 14, 2021 @06:03PM (#26137134)
You missed a big part: they did a big NAS run to make it work.
Re:DeepMind achieves SOTA image recognition with 8.7x faster... by rsfern on Sunday February 14, 2021 @08:58PM (#26138210)
Where did you see that they used NAS? Their preliminary results show it works even for the baseline model

they did a lot of manual hyperparameter optimization, and spend a fair amount of time unpacking the rationale for their choices, including a negative results section (!) in the appendix