Re:Every Model Learned by Gradient Descent Is Approximately ...
by abeppu on Sunday December 06, 2020 @11:54AM
(#25324679)
From a skim, I think one asterisk which may be a gap between the claim in the title and what's actually shown in the paper is that the theorem focuses on gradient descent-trained models which minimize a loss function which is the sum of loss L(y_i, y*_i) on points from a given dataset.
While that's clearly very broad, I think it _doesn't_ include things like GANs, where parts of the model produce fake data to train against.
[ Parent
| on HN ]
Re:Every Model Learned by Gradient Descent Is Approximately ...
by Flashtoo on Sunday December 06, 2020 @04:56PM
(#25327005)
The claim also applies to GANs as you can simply use a masking function to indicate which inputs were used for each optimization timestep, like the author suggests for stochastic gradient descent in remark 5.