I think there's something critical about the implicit data universe being considered in the test and training data, and in these randomized datasets. Memorizing elephants isn't necessarily a bad thing if you can be assured there are actually elephants in the data, or if your job is to reproduce some data that has some highly non-random, low entropy (in an abstract sense) features.

I think where the phenomenon in this paper, and deep double descent starts to clash with intuition, is the more realistic case where the adversarial data universe is structured, not random, but not conforming to the observed training target label alphabet (to borrow a term loosely from the IT literature). That is, it's interesting to know that these models can perfectly reproduce random data, but generalizing from training to test data isn't interesting in a real-world sense if both are constrained by some implicit features of the data universe involved in the modeling process (e.g., that the non-elephant data only differs randomly from the elephant data, or doesn't contain any nonelephants that aren't represented in the training data). So then you end up with this: https://www.theverge.com/2020/9/20/21447998/twitter-photo-pr...

I guess it seems to me there's a lot of implicit assumptions about the data space and what's actually being inferred in a lot of these DL models. The insight about SGD is useful, but maybe only underscores certain things, and seems to get lost in some of the discussion about DDD. Even Rademacher complexity isn't taken with regard to the entire dataspace, just over a uniformly random sampling of it -- so it will underrepresent certain corners of the data space that are highly nonuniform, low entropy, which is exactly where the trouble lies.

There's lots of fascinating stuff in this area of research, lots to say, glad to see it here on HN again.