In the pipelines you do some checks on statistical outliers/differences. Check the current training data against historical versions of the training set. If anything goes beyond some specified tolerances you highlight that for manual testing/checks.

Using the toy example from before, something like checking the sum of orders per customers in a month compared to the last N months. If the maximum per customer orders this month is 100x higher than any previous month then something has significantly changed in the data. May affect training, we need to investigate this.

If you've identified some statistical changes/differences, that's usually where someone needs to investigate in more depth. Train a dev model on the brand new training data. Pass multiple unseen test dataset(s) through it. What happens?

* Is global test accuracy up or down?

* Is robustness affected?

* Is the accuracy degrading for specific classes?

* How does this compare to drifts we've seen before?

Then you make decisions about whether you need to:

* exclude parts of the new training data?

* tweak some model hyperparameters?

* tweak the architecture of the model?

There's no single right answer on what to do at this point. This is the difficult and expensive bit of machine learning. It requires a lot of continuous experimentation even after you've got something running initially.