What I am complaining about is that the authors are confusing "what can be represented" by a neural network with "what can be easily learned" via SGD. I argued that the peak in test loss in those figures is observed becuse SGD struggles to find a solution that generalizes well, not because those networks are intrinsically less powerful (as the paper seems to imply).
Yeah... that's why the paper is a good contribution contrary to what you're saying. Not sure why you're repeating this information.
> What I am complaining about is that the authors are confusing "what can be represented" by a neural network with "what can be easily learned" via SGD. I argued that the peak in test loss in those figures is observed becuse SGD struggles to find a solution that generalizes well, not because those networks are intrinsically less powerful (as the paper seems to imply).
I mean.. those models are by definition less powerful as they have less parameters. The (to me) main point of the paper is to point out a symptom which is interesting. Their explanation for the symptom being (maybe) wrong doesn't detract from the important work of showing the symptom exists.
A legitimate criticism would be that there have been earlier papers showing the same thing.
Sidenote: It seems having a larger model does make it easier for SGD to find good solutions[1]
Its one of those papers where if you had a real handle on how everything works, wasnt surprising. But it was more a rebuke on some cargo cult ideas.
Lots of impactful papers have that form, eg., Gettier's paper on justified true belief.
Exactly because you select a single model and you have more models that underestimate the true (read test) error, you *may* be more likely to get models with worse performance on the test set compared to using a smaller network exactly because in the smaller network you are searching within a smaller distribution.
Though personally I do find a lot this modern "experimental" research quite hokey. I don't think this is something academics should be getting research funding to pursue. This is engineers building intuition about how to tune their product.
This is maybe true in the limit of infinite data, but not true in any practical sense, and I don't think it has anything to do with SGD. E.g. polynomial basis functions also have this property, but you can't use an arbitrarily large polynomial order or you'll eventually overfit. You can get a closed-form solution for polynomial regression problems, so no SGD involved.
All the paper shows is (1) stupid ways of counting model complexity and (2) that gradient descent is flawed. Nobody in their right mind believes that increasing the number of hidden neurons can result in a network that is worse on the test set. Since the bigger network contains the smaller network, it is perfectly capable of achieving the same performance, so the only reason why this does not happen is that SGD cannot find it. But of course "SGD cannot always find good solutions" is surprising to nobody, so let's just shit on decades of serious work to get our little paper out.
Sorry for the rant.