News for hackers. Stuff that matters.
Re:What Is Machine Learning Anyway? by amelius on Thursday July 07, 2022 @02:18AM (#32010536)
Can anyone explain: why is the Adam optimizer so unreasonably effective? Is anyone even using a different optimizer anymore?
Re:What Is Machine Learning Anyway? by Yenrabbit on Thursday July 07, 2022 @03:28AM (#32010912)
A good run-down of the different algorithms: https://ruder.io/optimizing-gradient-descent/ I think people favour adaptive learning rate options like Adam in practice since they generally do seem to perform well, and are often less sensitive to initial conditions and the exact hyper-parameters used. There will always be people who like to test N optimizers with parameter sweeps to squeeze a tiny bit of extra performance out, but for the rest of us the default Adam or AdamW options and good, unobjectionable choices :)
Re:What Is Machine Learning Anyway? by NavinF on Thursday July 07, 2022 @02:50AM (#32010692)
It’s really hard to compare optimizers. Common architectures and default hyperparameters were discovered alongside Adam so you’d have to redo a bunch of sweeps if you wanted a “fair” comparison. In practice this doesn’t really matter and everyone just uses Adam. If you had infinite compute, you’d try every combo and select the one with the best results.
Re:What Is Machine Learning Anyway? by locuscoeruleus on Thursday July 07, 2022 @04:23AM (#32011273)
Adam was very effective when it got introduced so it was widely adopted. Since then only models that work well with Adam have made it from the idea stage to actually working. I think there's reason to believe we have over fit our model architectures to our loss functions and optimizers.
Re:What Is Machine Learning Anyway? by jonbaer on Thursday July 07, 2022 @02:40AM (#32010647)
What is the current SOTA? Demon Adam? https://arxiv.org/pdf/1910.04952v4.pdf
Re:What Is Machine Learning Anyway? by yobbo on Thursday July 07, 2022 @03:57AM (#32011088)
Adam functions as a low-pass filter (and/or "compressor") on the gradient. It filters out "noise", which is "wild" during start of training.

This is basically what all "optimizers" achieve in various ways, including momentum.

Adam uses several times more memory, and is slower, than momentum or just SGD. That's reason to not use if not needed.