Re:Too much efficiency makes everything worse
by kqr on Friday November 11, 2022 @05:24AM
(#33558998)
I think this is a difficult concept for many without statistical training. The fact that different outcomes can be "the same" from a practical perspective.
Re:Too much efficiency makes everything worse
by jaschasd on Friday November 11, 2022 @10:00AM
(#33561596)
A gentle note that an incomplete piece of a goal (e.g. a loss function computed on a subset of the data) is a proxy for the full goal (e.g. the loss function on the full dataset).
Similarly, concept drift can be a source of overfitting -- the objective you care about is the one after the concept drift occurred, but the objective you trained on is the one from before the concept drift. (Here's a scholar search for papers where the two concepts co-occur: https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&as_vis... )
And the reply also leaves me with a similar impression:
> your training objective can be different at train and test time
But this is not overfitting, this is concept drift, a very different and well-defined thing in ML.
> the datapoints you use can be different at train and test time
Both train and test data came from the same population. They are just different incomplete random samples.
I guess what I am getting at - overfitting happens because we know we are training a model on an incomplete representation of the whole. But that representation is not a proxy, as suggested in the article - it is not slightly different to the goal. It's an incomplete piece of the goal.