As currently the only person doing data science things for the team, I’ll test to make sure changes I make to model/feature engineering/etc result in a better model. We’re not constantly, constantly retraining our models, because our incoming data and behaves the same. We’ve had the same model in prod for 4 months now; we don’t have any pressing issues with its predictions, and looking through the logs of what the input was the the output, it’s still performing as expected, so we’ll probably leave it longer.

I see, so how do you measure the difference between the incoming data and your training data?

Also, it looks like you have a very low volume of predictions?