News for hackers. Stuff that matters.
Re:A New Lens on Understanding Generalization in Deep Learning by woopwoop on Saturday March 20, 2021 @11:47PM (#26527961)
I agree with the philosophical conclusions here (e.g. I take their point on data augmentation to basically be that the name "data augmentation" is well chosen - it enlarges the restricted real world). I'm confused by the setup though. The training algorithm seeks to minimize the training loss on, in one case, the CIFAR-10 distribution D, and in the second case on D', their "ideal world" distribution. They find that these two tasks look very similar. I don't see what conclusions you can draw from this except that they did a good job training their generative model and D' is indeed very close to D, and so minimizers for the training loss on D' will look similar to those for D. (Close can mean something like the distributions D|_{label = c} and D|_{label=c'} are close in Earth-mover distance in some appropriate embedding space).
Re:A New Lens on Understanding Generalization in Deep Learning by antipaul on Sunday March 21, 2021 @01:07PM (#26531927)
Indeed. Isn’t that the whole point of the 100 years or whatever of statistical inference, where D is a sample and is representative of D’?