News for hackers. Stuff that matters.
Re:Are deep neural networks dramatically overfitted? (2019) by Nimitz14 on Monday April 05, 2021 @11:16AM (#26700128)
> And yet that is not what is observed in practice. See figures 1 and 2 of that paper

Yeah... that's why the paper is a good contribution contrary to what you're saying. Not sure why you're repeating this information.

> What I am complaining about is that the authors are confusing "what can be represented" by a neural network with "what can be easily learned" via SGD. I argued that the peak in test loss in those figures is observed becuse SGD struggles to find a solution that generalizes well, not because those networks are intrinsically less powerful (as the paper seems to imply).

I mean.. those models are by definition less powerful as they have less parameters. The (to me) main point of the paper is to point out a symptom which is interesting. Their explanation for the symptom being (maybe) wrong doesn't detract from the important work of showing the symptom exists.

A legitimate criticism would be that there have been earlier papers showing the same thing.

Sidenote: It seems having a larger model does make it easier for SGD to find good solutions[1]

[1] https://www.youtube.com/watch?v=kcVWAKf7UAg

There are no comments on this comment yet.