News for hackers. Stuff that matters.
Re:A New Lens on Understanding Generalization in Deep Learning by gwern on Sunday March 21, 2021 @12:21PM (#26531524)
n = 88 billion is larger than I can think of for any DL research I've read about. Can you (ahem) enlarge on that?
Re:A New Lens on Understanding Generalization in Deep Learning by chroem- on Sunday March 21, 2021 @01:21PM (#26532048)
It's an ultra high resolution sales forecast model for a national retailer. It's made somewhat more manageable by the fact that the data is 1D instead of 2D though.
Re:A New Lens on Understanding Generalization in Deep Learning by gwern on Monday March 22, 2021 @10:12AM (#26541023)
I wonder if that's an important difference? Sales data seems far more noisy and less informative than all of the scaling research datasets I can think of like images or text. It could also just be that you had some minor flaw causing underfitting in your approach like using too-small models which aren't compute-optimal or too much regularization (a critical part of scaling papers is removing standard regularizers like dropout to avoid hobbling the model).
Re:A New Lens on Understanding Generalization in Deep Learning by chroem- on Tuesday March 23, 2021 @02:46AM (#26551824)
I've found that it's surprisingly insensitive to the amount of regularization applied, however it definitely does warrant further experimentation. Also, I expect that one other distinction vs. image data is much higher correlation between samples in the dataset.