Yup. I learned this the hard way on the last model I trained at scale. It was evaluating fit between two heterogeneous classes I sampled training & test split off of a large time window and got to work. It performed extremely well on test & training. Too good.

I pulled a third sample from a completely different time window and it performed terribly.

It turned out that both datasets were dominated by class A being sorted into always selecting great fit or poor fit, so the ML model learned to memorize the class A instances.

This problem when away when I subselected down to only instances of class A that had examples of both good fit and poor fit.