In the pipelines you do some checks on statistical outliers/differences. Check the current training data against historical versions of the training set. If anything goes beyond some specified tolerances you highlight that for manual testing/checks.
Using the toy example from before, something like checking the sum of orders per customers in a month compared to the last N months. If the maximum per customer orders this month is 100x higher than any previous month then something has significantly changed in the data. May affect training, we need to investigate this.
If you've identified some statistical changes/differences, that's usually where someone needs to investigate in more depth. Train a dev model on the brand new training data. Pass multiple unseen test dataset(s) through it. What happens?
* Is global test accuracy up or down?
* Is robustness affected?
* Is the accuracy degrading for specific classes?
* How does this compare to drifts we've seen before?
Then you make decisions about whether you need to:
* exclude parts of the new training data?
* tweak some model hyperparameters?
* tweak the architecture of the model?
There's no single right answer on what to do at this point. This is the difficult and expensive bit of machine learning. It requires a lot of continuous experimentation even after you've got something running initially.
https://ppml.dev/design-code.html#data-debt
https://ppml.dev/troubleshooting-code.html#troubleshooting-d...
Probably a very over simplified example below. Because data doesn’t usually drift in this obvious way. It’s usually more subtle and happening over a longer period.
My model is learning on my business data of orders over time.
People keep ordering every day. but usually in small amounts.
But today we got a new customer and they put in monthly orders which are 1000x larger than all other combined.
They are going to keep making orders for the next year or so. At which point they stop ordering from us.
Two data drift “episodes” here:
1. When we get the new customer. We’ve now got an outlier. They aren’t like all the other customers. How’s will the model react to this when being trained? Will it skew the output? Do we exclude the new customer in training data? Or do we change the model to account for them?
2. When that customer stops ordering after a year. Now the outlier is gone. But maybe we changed some model settings and tweaked it a bit to account for it. Now we need to account for that customer not being around anymore.
Data drift is a big PITA.