Too much efficiency makes everything worse

Re:Too much efficiency makes everything worse by jaschasd on Friday November 11, 2022 @12:43PM (#33563993)

(the plot shows extreme overfitting with a 10 parameter model, and interpolation with a 10,000 parameter model)

[ Parent | on HN ]

Re:Too much efficiency makes everything worse by jaschasd on Friday November 11, 2022 @01:12PM (#33564444)

I can't reply directly -- is there a maximum thread depth, or a maximum conversation depth?

Anyway -- I wanted to apologize for misreading -- I missed the parenthetical "interpolation" in your comment. I think we are both interpreting the plot the same way.

In terms of your comment about anecdotal evidence -- are you talking about the case where data and model size are increased jointly? If so, I agree, though I don't think that is any longer cleanly to do with double descent/overparameterization.

[ Parent | on HN ]

Re:Too much efficiency makes everything worse by cs702 on Friday November 11, 2022 @12:59PM (#33564241)

Interpolation == extreme overfitting.

Double descent phenomenon is what happens after interpolation.

RESPONDING TO YOUR LAST COMMENT (after reaching thread depth limit):

Think of it this way: Why and how does the model's performance continue to improve on previously unseen samples after the model has fully overfit (interpolated between) all training samples? Interpolation is not the end-point in training, but a temporary threshold after which models learn to generalize better, improving on interpolation. How is it that these models improve on interpolation?

[ Parent | on HN ]