Re:A New Lens on Understanding Generalization in Deep Learning
by chroem- on Sunday March 21, 2021 @01:07AM
(#26528262)
I've had the opportunity to train on an 88 billion example dataset, and while you do see diminishing returns with each additional sample, it is still important to try to cover as much of the dataset as possible. The limiting factor on training on datasets that large seems to be some sort of hysteresis overfit on the earlier samples that prevents the model from generalizing to the entire dataset. Normally it goes unnoticed, but there is still a small amount of overfit that accumulates the longer you train the model, even if every sample is new.
[ Parent
| on HN ]
Re:A New Lens on Understanding Generalization in Deep Learning
by ShamelessC on Sunday March 21, 2021 @04:12AM
(#26528898)
Thanks this insight helped me understand an issues I'm currently having with training actually!
Re:A New Lens on Understanding Generalization in Deep Learning
by gwern on Sunday March 21, 2021 @12:21PM
(#26531524)
n = 88 billion is larger than I can think of for any DL research I've read about. Can you (ahem) enlarge on that?
Re:A New Lens on Understanding Generalization in Deep Learning
by chroem- on Sunday March 21, 2021 @01:21PM
(#26532048)
It's an ultra high resolution sales forecast model for a national retailer. It's made somewhat more manageable by the fact that the data is 1D instead of 2D though.
Re:A New Lens on Understanding Generalization in Deep Learning
by gwern on Monday March 22, 2021 @10:12AM
(#26541023)
I wonder if that's an important difference? Sales data seems far more noisy and less informative than all of the scaling research datasets I can think of like images or text. It could also just be that you had some minor flaw causing underfitting in your approach like using too-small models which aren't compute-optimal or too much regularization (a critical part of scaling papers is removing standard regularizers like dropout to avoid hobbling the model).
Re:A New Lens on Understanding Generalization in Deep Learning
by chroem- on Tuesday March 23, 2021 @02:46AM
(#26551824)
I've found that it's surprisingly insensitive to the amount of regularization applied, however it definitely does warrant further experimentation. Also, I expect that one other distinction vs. image data is much higher correlation between samples in the dataset.