News for hackers. Stuff that matters.
Re:Understanding “Deep Double Descent” by lostmsu on Saturday December 07, 2019 @02:11PM (#21731977)
Can someone explain what is "interpolation threshold"? Both articles talk about it, but none defines.
Re:Understanding “Deep Double Descent” by preetum on Saturday December 07, 2019 @03:04PM (#21732261)
Hi, we define the interpolation threshold in Section 2 of the full paper (https://arxiv.org/abs/1912.02292) as the point when the "Effective Model Complexity" = # of train samples.

Where the "Effective Model Complexity (EMC)" of a model + training procedure (w.r.t an input distribution) is the maximum number of samples from the distribution that the model+training can fit to ~0 train error.

Our experiments are consistent with the hypothesis that the double-descent peak occurs when EMC = n; that is, when the model+training is just barely able to fit the train set.

Re:Understanding “Deep Double Descent” by joe_the_user on Saturday December 07, 2019 @03:10PM (#21732290)
It seems like a phenomena of this sort would have to depend on the data set you are dealing with. It might be true for "all typical data sets in important domains" but it would still seem like basic point would hold.
Re:Understanding “Deep Double Descent” by preetum on Saturday December 07, 2019 @03:20PM (#21732341)
Oh yes, absolutely. We only make claims for "natural distributions and models" (^).

It's almost certainly possible to break this by pathological choice of a data distribution or model/initialization/optimization scheme. But, I don't think this is interesting -- what I think is interesting is that this seems to hold true in real life, in "natural settings".

---

(^) Whatever "natural distributions" means... (which I think is a good research direction in itself).

Re:Understanding “Deep Double Descent” by jackson1372 on Saturday December 07, 2019 @07:59PM (#21733725)
Isn't the explanation for this that the world actually does work in some way or other and that it's not just infinite chaos and so if you keep throwing parameters at some problem, you will eventually stumble upon the "real" structure, but that's no guarantee of when that occurs, and with which parameters?
Re:Understanding “Deep Double Descent” by joe_the_user on Saturday December 07, 2019 @10:48PM (#21734233)
Well, the thing is that when one says "the world has structure", one is saying that there are variety of structures "out there", in the world.

But that doesn't mean there's a single structure determined by a single set of parameters. Quite possibly there are numerous structures with not-compatible parameter structures.

Moreover, common AI data sets share parameters in a fashion that isn't always obvious - most images on the web are photos taken by human photographers who tend to center their subject, effectively giving them different parameters than, say, security camera footage. IE, "normal data" may not mean what we imagine.