Re:Test machine learning the right way: regression testing
by mklond on Tuesday July 06, 2021 @09:00AM
(#27747919)
Thanks for your comment. The whole field of run-time monitoring is concerned with this problem. It's a tough one to crack when the distribution changes are subtle, but you can and should at least check simple data attributes for consistency.
This form of ML bug is the most challenging to catch. The true in-operation distribution is often unknown which makes testing for such bugs a very challenging problem. Any thoughts on this?