News for hackers. Stuff that matters.
Re:Schedule-Free Learning – A New Way to Train by rand0mwalk on Saturday April 06, 2024 @01:43PM (#39954025)
Is there an accompanying paper out there?
Re:Schedule-Free Learning – A New Way to Train by JieJie on Saturday April 06, 2024 @03:22PM (#39954722)
From the Related Work section (best guess):

Stochastic Weight Averaging (Izmailov et al 2018) https://arxiv.org/abs/1803.05407

Latest Weight Averaging (Kaddour 2022) https://arxiv.org/abs/2209.14981

Latest Weight Averaging? (Sanyal et al 2023) https://arxiv.org/abs/2311.16294

Cyclic Learning Rates (Portes et al 2022) https://arxiv.org/abs/2206.00832

Exponential Moving Average? (Zhanghan? et al 2019) https://arxiv.org/abs/1909.01804

Re:Schedule-Free Learning – A New Way to Train by mkl on Saturday April 06, 2024 @08:21PM (#39956956)
Those are other people's papers about other methods.
Re:Schedule-Free Learning – A New Way to Train by JieJie on Monday April 08, 2024 @03:33PM (#39972916)
My mistake. I misread the comment that it was looking for links to the included research and I went to find them.
Re:Schedule-Free Learning – A New Way to Train by miven on Saturday April 06, 2024 @03:07PM (#39954630)
I think the author of this method said it's coming in a month or so
Re:Schedule-Free Learning – A New Way to Train by blt on Saturday April 06, 2024 @02:57PM (#39954552)
+1, curious to see if the paper has a convergence rate proof for convex objectives.
Re:Schedule-Free Learning – A New Way to Train by cutkosky on Saturday April 06, 2024 @03:38PM (#39954826)
(an) author here: paper will likely be coming out in O(month). But, yes it turns out that the method is minimax optimal for stochastic convex optimization for a wide variety of parameter settings. Of course, minimax optimality alone does not fully explain empirical success - we've had minimax optimal algorithms for decades!
Re:Schedule-Free Learning – A New Way to Train by xpe on Saturday April 06, 2024 @04:40PM (#39955295)
This would have been witty IMO: "the paper will be out in O(negative_peer_reviews)"

> (an) author here: paper will likely be coming out in O(month)

Ug. I'm adding "O(month)" to my list of bootless metaphors.

Why? (1) Because in Big-O notation, O(month) would equal O(day), which is not the intended meaning in the comment above; (2) It is non-sensical; one would never say e.g. "the run-time of an algorithm is O(seconds)" -- we write some kind of input inside the parens, not the output

Anyhow, we already have the words roughly and about; e.g. "about a month".

Feel free to call me pedantic, but words matter.

Re:Schedule-Free Learning – A New Way to Train by goldemerald on Saturday April 06, 2024 @07:59PM (#39956829)
I thought it was a clever/nerdy way to say in the worst case it will be out in a month. I imagine they have an internal review they have to get through first, and it's not clear if that will be done next week or in May.
Re:Schedule-Free Learning – A New Way to Train by pigpang on Sunday April 07, 2024 @12:38AM (#39958220)
We can safely assume that approximate time needed to produce papers is t_approx = O(n), where n is number of papers. O(t_approx) makes no sense.
Re:Schedule-Free Learning – A New Way to Train by xpe on Monday April 08, 2024 @11:04AM (#39970498)
I think the above comment probably meant "where n is number of _pages_".
Re:Schedule-Free Learning – A New Way to Train by blt on Sunday April 07, 2024 @01:40PM (#39962330)
agree, just a good "sanity check" for first-order optimization algos.