News for hackers. Stuff that matters.

A comment on Are language models deprived of electric sleep?. Parent comment

Re:Are language models deprived of electric sleep? by benreesman on Saturday July 23, 2022 @02:12PM (#32205929)
When you’re churning through terrawords of training data and aren’t cross-validating against proportionally-sized hold out sets?

It’s not a certainty that you’re overfitting but the burden is to show otherwise.

Besides, if your corpus is asymptomatically everything, why wouldn’t fitting it perfectly fitting it be the goal?

It would probably be more accurate to say that the bias/variance tradeoff loses meaning as the training set goes to infinity.

Grapevine is that the big actors are going to 5x their flops at any price in the next three years.

What does validation loss even mean at that scale?

[ Parent | on HN ]
Re:Are language models deprived of electric sleep? by Veedrac on Sunday July 24, 2022 @04:01AM (#32211351)
> Besides, if your corpus is asymptomatically everything, why wouldn’t fitting it perfectly fitting it be the goal?

That's called Google, and it serves a different purpose. Perfectly learning the distributional properties of text instead is not overfitting, that's just fitting.

[ Parent | on HN ]