News for hackers. Stuff that matters.
Re:What Is Machine Learning Anyway? by yobbo on Thursday July 07, 2022 @03:57AM (#32011088)
Adam functions as a low-pass filter (and/or "compressor") on the gradient. It filters out "noise", which is "wild" during start of training.

This is basically what all "optimizers" achieve in various ways, including momentum.

Adam uses several times more memory, and is slower, than momentum or just SGD. That's reason to not use if not needed.

There are no comments on this comment yet.