News for hackers. Stuff that matters.
Re:Beyond Gradient Averaging in Parallel Optimization by fchaubard on Tuesday December 31, 2024 @10:11AM (#42559138)
I think about designing your ideal solver. What do you want in a solver. I want my solver to squeeze all the juice out of the train that it possibly can and no more. If your problem is complete noise, I don’t want my solver getting 100% train accuracy as all SGD methods I am aware of do and as the soup method likely would as well as I am not averaging memorization thetas. I want my solver to score 0% train accuracy as GAF does. There may be other ways of getting there as well.

There are no comments on this comment yet.